Baker was born in Groesbeck, Texas, the son of Edna (née McDonald) and Doyle Charles Baker. He attended the University of North Texas, and graduated with a business degree in 1958. Baker served a tour of duty in the United States Army. In 1964 he appeared on stage in Marathon '33 at the ANTA Theatre in New York City. His career had its roots in television, though he did appear in several movies, including a part in the 1967 film Cool Hand Luke. He appeared in many television series, graduating to featured guest roles in such series as The Big Valley (in which he played a Harvard-educated Native American with a penchant for fighting) and Mod Squad (where he appeared as an illiterate vending machine robber). He was the title character in the 1971 TV movie Mongo's Back in Town, starring Telly Savalas.
Physically impressive at nearly 6'3" tall (189 cm.), he was a natural for Westerns, both on the big and little screens. While working regularly on television on top shows like Bonanza and Gunsmoke, he appeared in supporting roles in such films Guns of the Magnificent Seven (1969) and Blake Edwards' Wild Rovers, but his film career didn't start to quicken until he scored the role of Steve McQueen's younger brother in Sam Peckinpah's Junior Bonner, a film about a contemporary rodeo cowboy, which was released in late 1972.
His breakthrough came with the 1973 film Walking Tall, directed by Phil Karlson. (He would also star in the filmmaker's final work, Framed, two years later.) Released in February as a regional exploitation picture, Walking Tall clicked with audiences and became an unexpected hit, picked up for national distribution with a new TV ad campaign using the slogan, "When was the last time you stood up and applauded a movie?" The film eventually racked up $23 million at the box office (equivalent to approximately $124.1 million adjusted for inflation). Baker's performance was praised by influential film critic Pauline Kael, but he turned down the sequel.
Later that year, his work in Charley Varrick helped solidify Baker's reputation. He was praised for a courageous and offbeat portrait of the sadistic hitman Molly. He also co-starred with Robert Duvall in the 1973 crime film The Outfit and starred in the 1974 adventure film Golden Needles.
Baker has given many outstanding performances in a career spanning four decades. In 1980, he became the first actor to receive $1 million to star in a television series, the short-lived Eischeid (known as "Chief Of Detectives" in the UK).
He was "The Whammer," a baseball player modeled after Babe Ruth, in the 1984 baseball drama The Natural that starred Robert Redford. In a scene, the Whammer takes three swings at pitches from the young Roy Hobbs to try to impress a mysterious woman they have met on a train.
In 1985, he portrayed the corrupt Chief Jerry Karlin in Fletch. In the UK, he played CIA agent Darius Jedburgh in the BBC Television drama serial Edge of Darkness. He was nominated for "Best Actor" by the British Academy Television Awards, losing to his co-star Bob Peck.
Martin Scorsese directed him as a private detective in 1991's Cape Fear, hired by a man (Nick Nolte) whose family is being threatened by a psychopathic ex-convict (Robert De Niro).
Baker received the Distinguished Alumni Award from the University of North Texas in 1994.
While actor Carroll O'Connor was undergoing heart bypass surgery, Baker took his place on the television series In the Heat of the Night. Baker appeared as Captain Tom Dugan, a retired police captain who filled in while O'Connor's character was away at a police convention.
More recently, he has had appearances in Joe Dirt, The Dukes of Hazzard and Strange Wilderness.
In 2009, Baker delivered another performance in The Cleaner on A&E, playing an alcoholic military veteran attempting to help a friend cope with the loss of his son. He hires William Banks (played by Benjamin Bratt) to help him start back down the road to sobriety.
Baker played "King" in 2012's Mud with Matthew McConaughey, Reese Witherspoon and Sam Shepard.
In 1987, Baker played the villainous arms dealer Brad Whitaker in the Bond film The Living Daylights, starring Timothy Dalton as 007. In 1995 and 1997, Baker returned to the series, this time playing a different character, CIA agent Jack Wade, in GoldenEye and Tomorrow Never Dies, with Pierce Brosnan as Bond. Baker is one of four actors to appear as both a Bond ally and a villain, the others being Charles Gray, who appeared as Henderson in You Only Live Twice and Ernst Stavro Blofeld in Diamonds Are Forever; Walter Gotell, who appeared as Morzeny, the SPECTRE Island trainer, in From Russia with Love and as General Gogol, the head of the KGB, in six films between 1977 and 1987; and Richard Kiel as Jaws, in The Spy Who Loved Me and Moonraker, assisting 007 in the latter film.