Vandiver was born in Canon in Franklin County in northeastern Georgia. He was the only child of Vanna Bowers and Samuel Ernest Vandiver, Sr. His mother had two children from a previous marriage, which ended with the death of her first husband. Vandiver's father was a prominent businessman, farmer, and landowner in Franklin County. Vandiver attended public schools in Lavonia and the Darlington School in Rome, Georgia. He graduated from the University of Georgia and the University of Georgia School of Law, both in Athens.
After stateside service as an officer in the United States Army Air Corps during World War II, he was elected in 1946 as mayor of Lavonia in Franklin County. That same year he supported Eugene Talmadge's candidacy for governor that year and then Herman Talmadge's claim to the office after Eugene's death.
In 1948, Talmadge appointed Vandiver to be the state's adjutant general. In 1954, Vandiver was elected lieutenant governor. He ran for governor in 1958 and promised to restore the state's image, which had been tarnished by the scandals under Governor Marvin Griffin under whom he had served in the second position. Vandiver was overwhelmingly elected. Oddly, he succeeded Griffin as both lieutenant governor and governor.
As governor, Vandiver cleaned up the corruption and mismanagement associated with the Griffin administration. He had pledged to defend segregation, using the campaign motto, "No, not one," meaning not one black child in a white school.
During the presidential election of 1960, Vandiver supported John F. Kennedy, but not fully. Vandiver favored "independent" electors. This led to the erosion of the Democratic party in the South, and southern resistance to the civil rights movement. In March 1960, Vandiver called "An Appeal for Human Rights", an article published in the Atlanta Constitution by black students at Spelman College, "an anti-American document" that "does not sound like it was written in this country". Vandiver worked behind the scenes with Kennedy and his brother Robert and ultimately played a role in obtaining the release of Martin Luther King, Jr. from jail. King had been arrested during a sit-in at Rich's in Atlanta on October 19, 1960.
Vandiver had changed his "No, not one" stance, and changed his views on segregation. Those urging him to change include Ivan Allen Jr. (later mayor of Atlanta), banker Mills B. Lane, Coca-Cola President Robert Woodruff, Griffin Bell (later a judge of the United States Court of Appeals for the Fifth Circuit and the United States Attorney General under U.S. President Jimmy Carter), and many others.
Following Kennedy's assassination, Vandiver said, "All the world is shocked and grieved at the death of our President. I am certain that all Georgians join together in sending our condolences to the grieved family."
Under Vandiver's administration, a United States District Court ordered the admission of two African-American students, Hamilton E. Holmes and Charlayne Hunter, to the University of Georgia. Despite his past support for segregation, Governor Vandiver did not resist the court order. He hence spared the University of Georgia from the national publicity associated with the opposition stands taken in 1962 by Governor Ross Barnett at the University of Mississippi and in 1963 by Governor George C. Wallace at the University of Alabama. After the desegregation of the University of Georgia, Vandiver successfully urged the Georgia General Assembly to repeal a recently passed law barring state funding to integrated schools. He also appointed banker John A. Sibley to head a state commission designed to prepare for the court-ordered school desegregation.
Though he had been pledged to maintain the County Unit System, a type of electoral college that had been employed to elect Georgia governors, Vandiver was trumped by the United States Supreme Court, which struck down the system as unconstitutional. He hence ordered the Democratic State Central Committee to conduct the 1962 primary by popular vote.
Vandiver's efficiency in running state government permitted a building program and the expansion of state services without tax increases. The state expanded its ports, encouraged tourism, promoted business and industry, expanded vocational-technical education, and authorized programs for the mentally ill.
In 1966, Vandiver was initially a candidate for governor and had been expected to compete with another former governor, Ellis Arnall. However, Vandiver withdrew for health reasons. When the Democratic nomination went not to Arnall but to the Atlanta businessman Lester Maddox, a strong segregationist, the Democrat Vandiver endorsed in the general election the Republican nominee, U.S. Representative Howard "Bo" Callaway, then of Pine Mountain. Maddox was ultimately elected by the Georgia legislature after election returns failed to produce a winner by majority vote.
Had Vandiver's health permitted him to run for governor in 1966, Callaway would have instead sought reelection to the U.S. House. When Vandiver looked like a potential Democratic nominee, Callaway asked William R. Bowdoin, Sr. (1913-1996), an Atlanta banker and civic figure who had chaired a commission on state government reorganization, to run as a Republican gubernatorial candidate. Oddly, Carl Sanders, the term-limited governor, asked Bowdoin to run that year as a Democrat.
In 1972, at the age of fifty-four, Vandiver ran for the United States Senate for a full term to replace his wife's uncle, veteran Senator Richard Russell, Jr., who had died in office in 1971. Vandiver finished third behind Sam Nunn and appointed Senator David H. Gambrell in the Democratic primary election. Thereafter as the Democratic nominee, Nunn defeated the Republican Fletcher Thompson, an Atlanta-area U.S. representative even as Richard M. Nixon was sweeping Georgia in the presidential election against the Democrat George S. McGovern.
In his final years, he would express regret at his earlier segregationist positions. "I said a lot of intemperate things back then that I now have to live with," he said in 2002. "All I can say now is that you are of your time."
Vandiver was married to Betty Russell, a niece of Senator Russell, who had also served earlier as governor. Russell was popular and powerful in Georgia and helped to promote his nephew-in-law's career.
Vandiver was a nephew by marriage of Judge Robert Lee Russell and grandson-in-law of Judge Richard Russell, Sr. For information, see Russell family.
Ernest Vandiver died on February 21, 2005, at the age of eighty-six at his home in Lavonia, Georgia. In addition to his wife, he was survived by three children: Samuel Ernest "Chip" Vandiver, III; Vanna Elizabeth (named for her paternal grandmother and mother) Vandiver; and Jane Brevard Vandiver, who as Jane V. Kidd was elected as a member of the Georgia House of Representatives from Athens and in 2007 as the chairman of the Georgia Democratic Party.
The stretch of I-85 through Franklin County, Georgia, is named "Ernest Vandiver Highway" in his memory. Vandiver had worked to make sure the highway traversed Franklin County, instead of proceeding further north as originally planned.
On September 26, 2008, the University of Georgia dedicated a residence hall in the East Campus Village to Vandiver.