Agee was born in Knoxville, Tennessee, to Hugh James Agee and Laura Whitman Tyler, at Highland Avenue and 15th Street, which was renamed James Agee Street in 1909, in what is now the Fort Sanders neighborhood. When Agee was six, his father was killed in an automobile accident. From the age of seven, Agee and his younger sister, Emma, were educated in several boarding schools. The most prominent of these was located near his mother's summer cottage two miles from Sewanee, Tennessee. Saint Andrews School for Mountain Boys was run by the monastic Order of the Holy Cross affiliated with the Episcopal Church. It was there that Agee's lifelong friendship with Episcopal priest Father James Harold Flye, a history teacher at St. Andrew's, and his wife Grace Eleanor Houghton began in 1919. As Agee's close friend and mentor, Flye corresponded with him on literary and other topics through life and became a confidant of Agee's soul-wrestling. He published the letters after Agee's death. The New York Times Book Review pronounced The Letters of James Agee to Father Flye (1962 ) as "comparable in importance to Fitzgerald's 'The Crackup' and Thomas Wolfe's letters as a self-portrait of the artist in the modern American scene."
Agee's mother married St. Andrew's bursar Father Erskine Wright in 1924, and the two moved to Rockland, Maine. Agee went to Knoxville High School for the 1924–1925 school year, then traveled with Father Flye to Europe in the summer, when Agee was sixteen. On their return, Agee transferred to a boarding school in New Hampshire, entering the class of 1928 at Phillips Exeter Academy. Soon after, he began a correspondence with Dwight Macdonald.
At Phillips Exeter, Agee was president of The Lantern Club and editor of the Monthly where his first short stories, plays, poetry and articles were published. Despite barely passing many of his high school courses, Agee was admitted to Harvard University's class of 1932. There Agee took classes taught by Robert Hillyer and I. A. Richards; his classmate in those was the future poet and critic Robert Fitzgerald, with whom he would eventually work at Time. Agee was editor-in-chief of the Harvard Advocate and delivered the class ode at his commencement.
After graduation, Agee was hired by the Time Inc. as a reporter, and moved to New York City, where he wrote for Fortune magazine in 1932-1937, although he is better known for his later film criticism in Time and The Nation. In 1934, he published his only volume of poetry, Permit Me Voyage, with a foreword by Archibald MacLeish.
In the summer of 1936, during the Great Depression, Agee spent eight weeks on assignment for Fortune with photographer Walker Evans, living among sharecroppers in Alabama. While Fortune did not publish his article, Agee turned the material into a book entitled Let Us Now Praise Famous Men (1941). It sold only 600 copies before being remaindered. Another manuscript from the same assignment discovered in 2003, titled Cotton Tenants, is believed to be the essay submitted to Fortune editors. The 30,000 word text, accompanied by photographs by Walker Evans, was published as a book in June 2013. John Jeremiah Sullivan writes in the Summer 2013 issue of BookForum that, "This is not merely an early, partial draft of Famous Men, in other words, not just a different book; it’s a different Agee, an unknown Agee. Its excellence should enhance his reputation." A significant difference between the works is the use of original names in Cotton Tenants; Agee assigned fictional names to the subjects of Famous Men in order to protect their identity.
Agee left Fortune in 1937 while working on a book, then, in 1939, he took a book reviewing job at Time, sometimes reviewing up to six books per week; together, he and his friend Whittaker Chambers ran "the back of the book" for Time. In 1941, he became Time's film critic. In 1942-1948, he worked as a film critic for The Nation. Agee was an ardent champion of Charlie Chaplin's then unpopular film Monsieur Verdoux (1947), since recognized as a film classic. He was also a great admirer of Laurence Olivier's Henry V and Hamlet, especially Henry V. Agee on Film (1958) collected his writings of this period.
In 1948, Agee quit his job to become a freelance writer. One of his assignments was a well-received article for Life Magazine about the silent movie comedians Charles Chaplin, Buster Keaton, Harold Lloyd and Harry Langdon. The article has been credited for reviving Keaton's career. As a freelancer in the 1950s, Agee continued to write magazine articles while working on movie scripts; he developed a friendship with photographer Helen Levitt.
Agee's career as a movie scriptwriter was curtailed by his alcoholism. Nevertheless, he is one of the credited screenwriters on two of the most respected films of the 1950s: The African Queen (1951) and The Night of the Hunter (1955).
His contribution to Hunter is shrouded in controversy. Some critics have claimed the published script was written by the film's director Charles Laughton. Reports that Agee's screenplay for Hunter was incoherent have been proved false by the 2004 discovery of his first draft, which although 293 pages in length, is scene for scene the film which Laughton directed. While not yet published, the first draft has been read by scholars, most notably Professor Jeffrey Couchman of Columbia University. He credited Agee in the essay, "Credit Where Credit Is Due." Also false were reports that Agee was fired from the film. Laughton renewed Agee's contract and directed him to cut the script in half, which Agee did. Later, apparently at Robert Mitchum's request, Agee visited the set to settle a dispute between the star and Laughton. Letters and documents located in the archive of Agee's agent Paul Kohner bear this out; they were documented by Laughton's biographer Simon Callow, whose BFI book about The Night of the Hunter set this part of the record straight.
Soon after graduation from Harvard, he married Olivia Saunders (aka "Via") on January 28, 1933; they divorced in 1938. Later that same year, he married Alma Mailman. They divorced in 1941, and Alma moved to Mexico with their year-old son Joel, to live with Communist politician and writer Bodo Uhse.
Agee began living in Greenwich Village with Mia Fritsch, whom he married in 1946. They had two daughters, Julia and Andrea, and a son John. In 1951 in Santa Barbara, Agee, a hard drinker and chain-smoker, suffered a heart attack; on May 16, 1955, Agee was in New York City when he suffered a fatal heart attack in a taxi cab en route to a doctor's appointment. He was buried on a farm he owned at Hillsdale, New York, property still held by Agee descendants.
During his lifetime, Agee enjoyed only modest public recognition. Since his death, his literary reputation has grown. In 1957, his novel A Death in the Family (based on the events surrounding his father's death) was published posthumously and in 1958 won the Pulitzer Prize for fiction. In 2007, Dr. Michael Lofaro published a restored edition of the novel using Agee's original manuscripts. Agee's work had been heavily edited before its original publication by publisher David McDowell.
Agee's reviews and screenplays have been collected in two volumes of Agee on Film. There is some dispute over the extent of his participation in the writing of The Night of the Hunter.
Let Us Now Praise Famous Men has grown to be considered Agee's masterpiece. Ignored on its original publication in 1941, the book has since been placed among the greatest literary works of the 20th century by the New York School of Journalism and the New York Public Library. It was the inspiration for the Aaron Copland opera The Tender Land. David Simon, journalist and creator of acclaimed television series The Wire, credited the book with impacting him early in his career and informing his practice of journalism.
The composer Samuel Barber set sections of "Descriptions of Elysium" from Permit Me Voyage to music, composing a song based on "Sure On This Shining Night." In addition, he set prose from the "Knoxville" section of A Death in the Family in his work for soprano and orchestra entitled Knoxville: Summer of 1915. "Sure On This Shining Night" has also been set to music by composers René Clausen, Z. Randall Stroope and Morten Lauridsen.
In late 1979 the filmmaker, Ross Spears, premiered his film AGEE: A Sovereign Prince of the English Language, which was later nominated for an Academy Award for Best Documentary Feature and was awarded a Blue Ribbon at the 1980 American Film Festival. AGEE featured James Agee's friends, Dwight Macdonald, Robert Fitzgerald, Robert Saudek, and John Huston, as well as the three women to whom James Agee had been married. In addition, Father James Harold Flye was a featured interviewee. President Jimmy Carter speaks about his favorite book, Let Us Now Praise Famous Men.
The Man Who Lives Here Is Loony, a one-act play by Knoxville-based songwriter and playwright RB Morris, takes place in a New York apartment during one night in Agee's life. The play has been performed at venues around Knoxville, and at the Cornelia Street Cafe in Greenwich Village.