Gardenia was born Vincenzo Scognamiglio in Naples, Italy, the son of Elisa (Ausiello) and Gennaro Gardenia Scognamiglio. When he was two years old, the family immigrated to the United States and settled in Brooklyn, New York City, New York.
His father established an acting troupe that presented Italian-language melodramas. As a child, he performed in the troupe in Italian-American neighborhoods in and around New York City, having later said, "the titles changed, but they were usually about a son or daughter who gets in trouble, runs away, then begs forgiveness". He debuted in the company at age five, portraying a shoeshine boy. He graduated to character roles while still a teenager. He remained a member of the company until 1960, five years after his first English-speaking role on Broadway.
Gardenia played a small role in the film The House on 92nd Street and bit parts in other films, including Cop Hater and A View From the Bridge. His first English-speaking role was in 1955, as a pirate in the Broadway play In April Once. The following year, at age 36, he appeared as Piggy in his Off-Broadway debut in The Man with the Golden Arm. He described his role in the film Little Murders as a "turning point". He won Obie Awards in 1960 and 1969.
A life member of The Actors Studio, Gardenia won a Tony Award for Best Featured Actor in a Play in 1972 for his performance in The Prisoner of Second Avenue, opposite Peter Falk. In 1979, he was nominated for Best Actor in a Musical for his performance in Ballroom.
In film, he was twice nominated for an Academy Award for Best Supporting Actor for his performances in Bang the Drum Slowly and Moonstruck.
In television, Gardenia won the 1990 Emmy Award for his performance in Age-Old Friends, with Hume Cronyn. Among his best remembered TV roles is his portrayal of Archie Bunker's neighbor Frank Lorenzo on All in the Family (1973–74) and as J. Edgar Hoover in the miniseries Kennedy (1983). He also played in an episode of Voyage To The Bottom Of The Sea titled "Escape From Venice."
In December 1992, Gardenia was in Philadelphia to perform in the stage production of the Tom Dulack comedy Breaking Legs. He was beginning a three-week run as restaurant owner Lou Garziano in the off-Broadway hit at the Forrest Theatre. It was a role he had performed since the show's New York opening in May 1991.
Around 1 a.m. on December 9, 1992, hours after the final preview performance, Gardenia had returned to his Philadelphia hotel after dining with stage director John Tillinger, producer Elliot Martin, and cast members. According to Martin, Gardenia showed no signs of illness, adding, "It was just a jolly evening." According to authorities, when Gardenia failed to appear the next morning for a radio interview to promote the play's run, press representative Irene Gandy and cast member Vince Viverito became alarmed. When they arrived at Gardenia's hotel room, there was no answer. The hotel sent an engineer who opened the door and Gardenia was discovered dead of a heart attack, dressed and clutching the telephone. He was 72.
That evening, in the theatrical tradition of "the show must go on" and just hours after Gardenia's death, the play's official opening took place. The company dedicated its opening performance to Gardenia's memory. Harry Guardino assumed Gardenia's role as the restaurant owner.
Gardenia is interred in Saint Charles Cemetery in Farmingdale, Long Island, New York, along with his parents Elisa (1901–1967) and Gennaro Gardenia Scognamiglio (1896–1965). Gardenia never married and was survived by his brother, Ralph. A section of 16th Avenue in the Bensonhurst neighborhood of Brooklyn, New York, where he resided until his death, bears the secondary name of Vincent Gardenia Boulevard in his honor.