Carmichael was born in Columbia in Marion County in southwestern Mississippi. After graduation from high school, from 1944 to 1945 and again from 1951 to 1953, he served in the United States Coast Guard. In 1950, he obtained a Bachelor of Science degree from Texas A&M University in College Station, Texas. In 1976, he was a fellow at the John F. Kennedy School of Government at Harvard University in Cambridge, Massachusetts. Beginning in 1961, Carmichael was a Volkswagen dealer in Meridian in Lauderdale County and a real estate developer. In 1968, Carmichael launched Missouth Properties, a commercial real estate firm in Meridian, since run by his son, Gilbert Scott Carmichael (born January 1964). Carmichael died at the age of eighty-eight of a heart attack at Anderson Regional Medical Center in Meridian.
In the 1963 Mississippi gubernatorial election, Carmichael supported Rubel Phillips, the first Republican to seek the state's highest office in decades. A former member of the Mississippi Public Service Commission from Corinth in northeastern Mississippi, Phillips challenged the Democratic nominee, Paul B. Johnson, Jr., the son of an earlier governor, Paul B. Johnson, Sr. In a civic club debate in Meridian, Carmichael remarked that it was:
peculiar to defend something I have always taken for granted. ... Who, a few years ago, would have thought it possible to have reached the point ... where it is necessary to defend the human right to a choice? ... I fear that the state Democratic Party is unwittingly being used as a tool for the same goals of the national party. ... Today the majority of whites in this state are not one-party Democrats but true independents - and they are glad that there is the beginning of a second party - so that they can have a real choice.
Carmichael ran unsuccessfully for the Mississippi State Senate in both 1966 and 1967, the latter on the statewide Republican ticket headed for a second time by Rubel Phillips. In 1971, Carmichael considered a race for lieutenant governor, but Republicans that year fielded no statewide candidates as the Democrat Bill Waller defeated the African-American Independent Charles Evers, a civil rights activist and the mayor of Fayette, in the general election for governor.
Carmichael was a delegate to the 1972 Republican National Convention, which met in Miami Beach, Florida, to re-nominate President Nixon and Vice President Agnew Carmichael was the Republican nominee in 1972 for the United States Senate against the veteran Democrat James O. Eastland. In the Republican primary, Carmichael easily defeated the African-American civil rights figure James H. Meredith. In the fall campaign, President Nixon directed that Carmichael not attend a Republican rally in the capital city of Jackson hosted by Agnew, who endorsed two successful Republican candidates for the United States House of Representatives, Thad Cochran and Trent Lott. Instead Carmichael was humiliated and had to remain at Republican headquarters two blocks away from the festivities. One of Eastland's press spokesmen was Larry Speakes, later the White House acting press secretary in the Ronald W. Reagan administration.
Carmichael finished the race against Eastland with 249,779 votes (38.7 percent). Prentiss Walker of Smith County, the first Mississippi Republican to serve in the U.S. House of Representatives in the 20th century, ran as an Independent in the Senate general election and drew 14,662 votes (2.3 percent). Walker had run as a strongly conservative Republican against Eastland in 1966 and claimed that the veteran senator was too close to Democratic President Lyndon B. Johnson and did little as chairman of the Senate Judiciary Committee to block the confirmation of liberal judges prone to integrationist rulings. Eastland in his last victory at the polls won the race with 375,102 (58.1 percent); he did not seek reelection in 1978 and was succeeded by Republican Congressman Thad Cochran.
In 1973, Carmichael's friend, Tom Stuart was elected as the first Republican mayor of Meridian, in which capacity Stuart sought with some success to pave streets and resolve downtown traffic problems.
In 1975 and 1979, Carmichael ran for governor against the Democratic nominees, Cliff Finch of Batesville and William Winter, having polled 45.1 and 38.9 percent of the vote, respectively. In the race against Finch, who openly appealed for black support but sounded like George Wallace of neighboring Alabama, Carmichael ran as a Moderate Republican. He promised economic development, the end of the cheap export of the state's natural resources, a new state constitution, ratification of the proposed Equal Rights Amendment to the United States Constitution, reduced punishment for possession of marijuana, and even the licensing of handguns.
Carmichael often appealed for African-American support before the conservative faction gained control of the fledgling Mississippi GOP. In 1976, Carmichael supported the nomination of Nixon's immediate successor, U.S. President Gerald R. Ford, Jr., rather than Ford's intra-party challenger, former Governor Ronald W. Reagan of California. This campaign brought Carmichael and the state party chairman Clarke Reed in conflict with the Republican finance chairman, W. D. "Billy" Mounger, an oilman and a Reagan partisan who had criticized the Nixon's snub of Carmichael in 1972 and had supported Carmichael in the 1975 gubernatorial race despite reservations over the liberal parts of the party platform. However, Mounger grew so disillusioned with Carmichael, whom he expected to defeat Finch, that he wrote a letter on election night renouncing any links to the candidate. Mounger specifically disagreed with Carmichael's support of the ERA, handgun registration, and a new constitution as evidence of "ineptitude" and "Teddy Kennedy appeal to liberals."
Former U.S. Representative Prentiss Walker, who had opposed Carmichael as an Independent in the 1972 U.S. Senate race, wrote a column in The Clarion-Ledger, the major newspaper in Jackson, which called Carmichael "a real discredit to all the true Republican principles." Walker said that Carmichael if elected would revise the constitution "under the direction of Senators Brooke and Javits", two liberal Republicans from Massachusetts and New York, respectively.
Eight years after his first gubernatorial campaign, Carmichael in 1983 polled 35.7 percent of the vote in his final statewide race against Democratic Lieutenant Governor Brad Dye. This time Carmichael ran on the Republican ticket headed by a former Democrat, Leon Bramlett of Clarksdale, whom Carmichael had defeated, 53-47 percent, in the preceding 1979 gubernatorial primary.
In 1973, Carmichael joined the National Highway Safety Advisory Committee and was chairman of the committee from 1974 to 1976, while he made his first race for governor. From 1976 to 1979, he was a member of the National Transportation Policy Study Commission.
From 1989 to 1993, Carmichael served in the administration of U.S. President George Herbert Walker Bush as the head of the Federal Railroad Administration within the United States Department of Transportation. He is a former chairman of Amtrak. Carmichael founded the Intermodal Transportation Institute at the University of Denver in Denver, Colorado. In his later years, he hence became more identified with transportation policy than Republican politics.
Carmichael advocated the construction of a railroad to link the ports of Mobile, Alabama and Pascagoula, Mississippi north to Lucedale and Waynesboro, Mississippi, and then join with the Meridian Southern Railroad line running through Quitman and to Meridian.
Carmichael died in Meridian early in 2016 at the age of eighty-eight.