Schuerholz was born in Baltimore, the son of John Schuerholz Sr., who played in the Philadelphia Athletics minor league system from 1937 to 1940. He is a graduate of the Baltimore City College High School, Towson University and Loyola University. While at Towson, Schuerholz applied for officer candidate school and was rejected, as he was partially deaf. Before his career in baseball, Schuerholz was a teacher at North Point Junior High in Baltimore. Upon leaving his teaching job, he was drafted by the United States Army to serve in the Vietnam War. After entering Major League Baseball with the Baltimore Orioles, Schuerholz joined the United States Army Reserve.
He donated $250,000 to Towson in 1999. This money was used to upgrade the school's baseball facility, which was named after Schuerholz.
Schuerholz's son, Jonathan, was selected by Atlanta in the eighth round of the 2002 MLB draft and played in the minor leagues until 2007. Jonathan retired from baseball in August 2007 to go back to Auburn University to complete his business degree. The younger Schuerholz, who finished his six-year career in the minor leagues with a .223 batting average, was named manager of the Rome Braves (Atlanta's Class-A minor league affiliate) in 2014. After the season, Jonathan was reassigned to the Braves front office to serve as an assistant player-development director.
The Baltimore Orioles hired Schuerholz in 1966 as a result of a letter Schuerholz wrote to team owner Jerold Hoffberger. Schuerholz worked under Frank Cashen, Harry Dalton, and Lou Gorman. In 1969, Major League Baseball expanded to Kansas City. Gorman and Schuerholz left for the Royals. Schuerholz was named general manager of the Royals during the 1981 offseason, and became Major League Baseball's youngest general manager at the time. Schuerholz built a strong relationship with Royals owner Ewing Kauffman, but left the team as it began to struggle. He joined the Braves in 1990, succeeding Bobby Cox who returned to the dugout to manage the team. The duo of Schuerholz and Cox produced an unprecedented run of success for the franchise, highlighted by the 1995 World Series Championship. On October 11, 2007, Schuerholz resigned as the Atlanta Braves general manager, but was promoted to club president, replacing Terry McGuirk. Schuerholz's top assistant Frank Wren was named the general manager. When Schuerholz stepped down as club president in March 2016, his duties were split between Derek Schiller, as president of business, and Mike Plant, as president of development.
Schuerholz has sent many assistants to general manager positions around the league, including Wren and Braves current GM John Coppolella. Dayton Moore, the Braves' former Director of Scouting and assistant GM under Schuerholz, has been GM of the Kansas City Royals since 2006, when he replaced Allard Baird.
In 2006, Schuerholz published a book, Built To Win, which chronicled his tenure with the Braves and some of his most important moves as a GM. Included in his book is a trade the Braves almost made with the Pirates in 1992. Had the deal gone through the Braves would have sent pitcher Alejandro Pena and outfielder Keith Mitchell to the Pirates in exchange for Barry Bonds.
On December 4, 2016, Schuerholz was elected into the National Baseball Hall of Fame and Museum in the Class of 2017.