Burgess was born in Providence, Rhode Island, the son of Thomas Burgess, a Rhode Island judge, and his wife, Mary (Mackie) Burgess. Burgess's older brother, Thomas Mackie Burgess, was mayor of Providence. His younger brother, Alexander Burgess, was also an Episcopal bishop. He graduated from Brown University in 1826, with the highest honors. After college, Burgess spent some time abroad from 1831-34 in Göttingen, Bonn, and Berlin. Bishop Alexander Viets Griswold admitted Burgess to deacon's orders, in Providence, June 10, 1834. He was ordained priest shortly thereafter, on November 2, 1834. He then became rector of Christ Church in Hartford, Connecticut.
Burgess was married in October 1846 to Sophia Kip. He was elected first bishop of Maine, early in October 1847, and consecrated in Christ Church, Hartford, that same month. He was the 49th bishop of the ECUSA, and was consecrated by bishops Philander Chase, Thomas Church Brownell, and Manton Eastburn. In 1850, shortly after his elevation to the Episcopate, the Burgesses' only child, Mary Georgianna, was born.
On removing to Maine, Burgess became the rector of the church in Gardiner, a place he retained until his death. Burgess joined the William Augustus Muhlenberg in the "Memorial Movement" (characterized by Muhlenberg as an "evangelical catholic" movement) in 1853. His ministry was of the style of Muhlenberg, Alonzo Potter, and Alexander Griswold, who were sometimes called high church evangelicals.
Burgess was one of the presenters of Bishop George Washington Doane of New Jersey, on charges concerning Doane's financial integrity. He was suffered a severe hemorrhage in July 1865, but still sailed for the West Indies in December by appointment of the house of bishops to visit Haiti in the interests of the church. He died at sea, near Port au Prince in 1866, of natural causes. Burgess's daughter died unmarried in 1873. His widow, Sophia, lived until 1907, never having remarried.
He authored several publications, including "The Book of Psalms, translated into English Verse" (1840); " Strife of Brothers," a poem (1844); "Pages from the Ecclesiastical History of New England between 1740 and 1840" (1847); "The Last Enemy" (1850); and "Sermon on the Christian Life" (1854)". After his death a volume containing his "Poems" was published, with an introduction by Arthur Cleveland Coxe (1868).