Fraser was born in Hinesville, Georgia on January 26, 1926 to Joseph B. Fraser Sr. and Pearl Collins. He preceded his brother, Charles Elbert Fraser, who was born in 1929. His father was a member of the Georgia National Guard at Fort Stewart and would become the base commander, serving in World War I, World War II and the Korean War and retired as a Lt. General. His father also owned and ran a logging company and the Fraser Supply Company which Joe would run before and immediately after college.
Growing up as the son of a base commander, Fraser had a horse at young age. He was once caught riding his horse through the Hinesville County Courthouse at night.
As a child, Fraser became ill with the Polio virus. At the time, many Doctors were prescribing the use of an iron lung, however, Fraser's Doctor prescribed exercise. He eventually recovered but with limited use of his left arm for the rest of his life.
He attended the University of Georgia and graduating in 1949 with a bachelor's degree in marketing. He was also a member of the Sigma Nu fraternity.
During World War II, Fraser enlisted in the Army, completing one handed push-ups to pass his physicals to hide the fact that he had a polio-weakened arm. He briefly served but did not deploy overseas.
The Fraser family owned the Fraser Lumber Company, founded in 1929, and the Fraser Supply company, founded in 1956. When Fraser's father left for World War II, Joe Jr. ran the lumber company. Much of the timber operation at the time was based on cutting naturally seeded forests, rather than forest replanting operation in common practice today. In 1949, in efforts to seek out new properties to timber, Fraser's father joined a group of lumber associates to buy a total of 20,000 acres of pine forest on Hilton Head Island's southern end for an average of nearly $60 an acre. In 1956, Fraser's younger brother Charles, who had recently graduated from Yale law, bought his father interest in the Hilton Head Company and created the Sea Pines Company. In the 1960s, Joe joined the Sea Pines Company as the first President of Sea Pines Homebuilders and later as President of Planning, Design and Construction and President of Sea Pines Plantation Company. From the mid-1960s to mid-1970s Mr. Fraser led the development of some of the most acclaimed residential and resort communities on the continent including Sea Pines Plantation, Hilton Head Plantation, Amelia Island Plantation and Kiawah Island.
Fraser and his development group were known for pioneering architecture and communities that blended with the environment, for conservation-minded construction and protection of wildlife preserves and green space.
When the Sea Pines Company was sold in the 1980s, Fraser turned his attention to new development and philanthropic opportunities. Under his leadership in the 1990s, The Heritage became a premier PGA Tour events providing more than $19 million to local and state charities through 2009.
Fraser married Carolyn Bexley and together they had four sons and one daughter. Their third son, West, is a well known artist in Charleston for plein air oil paintings. On July 29, 1987, Carolyn Bexley Fraser died of breast cancer. Fraser remarried Jean O'keefe from Augusta, GA and lived on Hilton Head Island until his death March 13, 2014 aged 88.