Warden was born John Warden Lebzelter Jr. in Newark, New Jersey, the son of Laura M. (née Costello) and John Warden Lebzelter, who was an engineer and technician.
He was of Pennsylvania Dutch (German) and Irish ancestry. Raised in Louisville, Kentucky, he was expelled from high school for fighting and eventually fought as a professional boxer under the name Johnny Costello. He fought in 13 bouts as a welterweight, but earned little money.
Warden worked as a nightclub bouncer, tugboat deckhand and lifeguard before joining the United States Navy in 1938. He was stationed for three years in China with the Yangtze River Patrol.
In 1941, he joined the United States Merchant Marine but he quickly tired of the long convoy runs, and in 1942 he moved to the United States Army, where he served as a paratrooper in the 501st Parachute Infantry Regiment, with the 101st Airborne Division in World War II. In 1944, on the eve of the D-Day invasion (in which many of his friends would die), Warden, now a Staff Sergeant, shattered his leg when he landed in a tree during a night-time practice jump in England. He spent almost eight months in the hospital recuperating, during which time he read a Clifford Odets play and decided to become an actor. Ironically, Warden would later portray a paratrooper from the 101st's rivals – the 82nd Airborne Division – in That Kind of Woman.
After leaving the military, he moved to New York City and studied acting on the G.I. Bill. He joined the company of the Dallas Alley Theatre and performed on stage for five years. In 1948 he made his television debut on the anthology series, The Philco Television Playhouse and Studio One. His first film role, uncredited, was in the 1951 film You're in the Navy Now, a film which also featured the screen debuts of Lee Marvin and Charles Bronson.
Warden appeared in his first credited film role in the 1951 in The Man with My Face. From 1952 to 1955, Warden appeared in the television series Mister Peepers with Wally Cox. In 1953, Warden was cast as a sympathetic corporal in From Here to Eternity. Warden's breakthrough film role was his performance as Juror No. 7, a salesman who wants a quick decision in a murder case, in 12 Angry Men.
Warden guest-starred in many television series over the years, including two 1960 episodes of NBC's The Outlaws, on Marilyn Maxwell's ABC drama series, Bus Stop, and on David Janssen's ABC drama, The Fugitive. He received a supporting actor Emmy Award for his performance as Chicago Bears coach George Halas in the television movie, Brian's Song, and was twice nominated for his starring role in the 1980s comedy/drama series Crazy Like a Fox.
Warden was nominated for Academy Awards as Best Supporting Actor for his performances in Shampoo and Heaven Can Wait. He also had notable roles in Bye Bye Braverman, All the President's Men, ...And Justice for All, Being There, Used Cars (in which he played dual roles), The Verdict, Problem Child and its sequel, as well as While You Were Sleeping, Guilty as Sin and the Norm Macdonald comedy Dirty Work. His final film was The Replacements in 2000, opposite Gene Hackman and Keanu Reeves.
Warden married French actress Vanda Dupre in 1958 and had one son, Christopher. Although they separated in the 1970s, the couple never divorced.
Warden suffered from declining health in his last years which resulted in his retirement from acting in 2000. He died of heart and kidney failure in a New York hospital on July 19, 2006, at the age of 85.