He was born in Ellon, Aberdeenshire, Scotland, and immigrated to the United States in 1839. He pursued a career in banking in Milwaukee, and founded the Marine Bank of Wisconsin.
Mitchell was president of the Chicago, Milwaukee and St. Paul Railway from 1864-1887. With fellow director Jeremiah Milbank (1818–1884) he built this railroad into one of the most profitable in the United States, and Mitchell was considered the wealthiest person in Wisconsin.
Mitchell represented Wisconsin's 1st congressional district in the Forty-second United States Congress. After redistricting he represented Wisconsin's 4th congressional district in the Forty-third United States Congress. He was nominated for Governor of Wisconsin in 1877, but he declined.
He was an avid curler, and helped popularize the sport in the United States. Mitchell helped found the Milwaukee Curling Club in the 1840s, and shortly before his death was elected Patron of the Grand National Curling Club.
Mitchell died in New York City and was buried at Forest Home Cemetery in Milwaukee.
Mitchell owned a mansion across the street from the Milwaukee County Courthouse, which is now the site of the Wisconsin Club.
Both Mitchell Park and the Historic Mitchell Street neighborhood were named in his honor, as was the city of Mitchell, South Dakota, incorporated in 1881.
In World War II, the United States liberty ship SS Alexander Mitchell was named in his honor.
The Mackie Building, which was constructed by Mitchell as an investment property, is listed on the National Register of Historic Places. Additionally, the Mitchell Building, which he also built, is listed on the National Register of Historic Places as well.
His papers, along with those of his son John, are in the archives of the Wisconsin Historical Society.
Mitchell was married to Martha Reed, sister of Harrison Reed, who served as Governor of Florida during Reconstruction. Mitchell's son, John L. Mitchell, was a Congressman and United States Senator, and his grandson, Billy Mitchell, was a United States Army officer prominent during the early days of military aviation.