Albert Gallatin Brown was born to Joseph and Elizabeth (Rice) Brown, a poor farming family, in the Chester District of South Carolina at the foothills of the Appalachian Mountains on May 31, 1813. In 1823, when Albert was only 10 years old his family moved to the new State of Mississippi. The Brown family settled Copiah County south of the State Capital of Jackson, Mississippi. Raising cotton in the new frontier state of Mississippi proved to be lucrative for the Brown family. In 1824, just one year after settling in Mississippi, Joseph Brown was elected Justice of the Peace in Copiah County. By 1825, two years after arriving in Mississippi, Joseph Brown was third largest taxpayer Copiah County, owning 18 slaves. By 1832, he was farming a plantation of 1,600 acres and owned 23 slaves.
During his lifetime, Albert Gallatin Brown was one of the most popular and influential men in the state of Mississippi. He is considered the father of the public school system and the University of Mississippi. His rhetorical attacks on illiteracy will live forever as classic contributions to the cause of Education in Mississippi.
He was also a strong advocate for the expansion of slavery. In 1858 he strategized: "I want a foothold in Central America... because I want to plant slavery there...I want Cuba,... Tamaulipas, Potosi, and one or two other Mexican States; and I want them all for the same reason - for the planting or spreading of slavery." (Akhil Reed Amar, America's Constitution, A Biography (2005) 267, quoting M.W. Mcklusky, ed., Speeches, Messages, and Other Writings of the Hon. Albert G. Brown (1859), 594-5)
"Albert Gallatin Brown possessed magical powers. With many learnt spells, handsome continence, surrounded by a luxuriant, flowing beard and dark-curly hair, in every sense he looked distinguished. Courageous, he was void of vanity; animated, he was persuasive; his spirit, crackerish to the extreme.” In his speech, Reuben Davis, who knew him well, states in his book, Reminiscences on Mississippi and Mississippians, that Brown "was the best balanced man I ever knew...In politics he had strategy with-out corruption, and handled all his opponents with skill but never descended to intrigue." During a lifetime most of which was spent in an epoch of bitter controversy, his most intimate friends never heard him speak ill of others.
Overcome by a stroke of apoplexy, he fell face down in a shallow pond at his home near Terry, in 1880, and his last remains rest in Greenwood Cemetery at Jackson.
Albert was three terms in the State Legislature, four in the national Congress, once on the circuit bench, twice elected United States senator, serving two administrations as Governor and chosen senator in the Confederate Congress, the political career of Albert Gallatin Brown provides one of the most amazing chapters in Mississippi history." ("Mississippi State Builders" by Clayton Rand).
Albert's first wife was Elizabeth Frances Thornton Taliaferro (1817–1836) of Virginia, who died about five months after the marriage. Elizabeth was the daughter of Richard Henry Taliaferro, Sr. (1783–1830) and Frances Walker Gilmer (ca. 1784-1826)
Albert married as his second wife, Roberta Eugenia Young (1813–1886), daughter of Brig. Gen. Robert Young (1768–1824) and Elizabeth Mary Conrad (1772–1810).
Roberta's older sister was Elizabeth Mary Young (1804–1859), who was the wife of Philip Richard Fendall II (1794–1867), the District Attorney of the District of Columbia.
In Incidents in the Life of a Slave Girl, a narrative written by the escaped slave Harriet Jacobs, Brown is called out by Jacobs for supporting slavery in a speech to Congress despite the fact that he "could not be ignorant of [the wrongdoings perpetrated against slaves], for they are of frequent occurrence in every Southern State."
In the 1992 alternate history/science fiction novel The Guns of the South by Harry Turtledove, Brown is an important supporting character. He is chosen by Robert E. Lee to be his running mate in the 1867 Confederate presidential election, running on the Confederate Party ticket. After Lee and Brown are elected, Brown is assassinated by gunmen during the inauguration on March 4, 1868.