Salerno was born and raised in East Harlem, New York. As a young man, he became involved in gambling, numbers, loansharking and protection rackets for the Lucky Luciano family, which later became the Genovese family. Salerno was a member of the 116th Street Crew, headed by Michael "Trigger Mike" Coppola. Salerno climbed the family ranks by controlling a possible million-dollar-a-year numbers racket operation in Harlem and a major loansharking operation. In 1948, Coppola fled to Florida to escape murder charges, and Salerno took over the crew. For the next four decades, he was one of the most powerful mobsters in America.
Unlike other mob bosses who were remote and reluctant to talk to outsiders, Salerno was very accessible. Mafiosi from Cleveland, Philadelphia, New England, Buffalo and other cities would visit Salerno to talk about various internal problems they wanted resolved. Salerno preferred a low-key existence and led an unpretentious life. He was never spotted at glitzy mob parties, nightclubs or other popular Mafia bistros. He even sent out Christmas cards with a picture of himself in pajamas on the front cover.
In 1959, Salerno was a secret financial backer of a heavyweight professional boxing title fight at New York's Yankee Stadium between Swedish boxer Ingemar Johansson and American boxer Floyd Patterson. No charges were filed against Salerno. Salerno divided his time between a home in Miami Beach, Florida, a 100-acre (0.40 km2) estate and horse farm in upstate Rhinebeck, New York, the Palma Boys Club in East Harlem, and his apartment in the upscale Gramercy Park section of Manhattan. He controlled S&A, a concrete contracting company—one of the two major concrete suppliers in Manhattan. Salerno served as consigliere, underboss, and acting boss of the Genovese family.
By the 1960s, Salerno controlled the largest numbers racket operation in New York, grossing up to $50 million per year. Many mobsters moved out of Harlem and East Harlem when they became predominantly Latino and African-American neighborhoods. However, Salerno kept his headquarters at the Palma Boys Social Club in East Harlem and continued to work in these areas. The FBI accused him of heading a bookie and loan shark network that grossed $1 million annually. Salerno hired Roy Cohn as his attorney. On April 20, 1978, Salerno was sentenced to six months in federal prison for illegal gambling and tax evasion charges. In early 1981, after his release from prison, Salerno suffered a mild stroke and retreated to his Rhinebeck estate to recuperate. At the time of his stroke, Salerno was Genovese underboss.
After Salerno's recovery from his stroke and the March 31, 1981 death of Genovese front boss Frank Tieri, Salerno succeeded him. Although law enforcement at the time thought that Salerno was the boss of the Genovese family, it was an open secret in New York Mafia circles that Salerno was merely a front man for the real boss, Vincent "the Chin" Gigante. For instance, Alphonse "Little Al" D'Arco, who later became acting boss of the Lucchese crime family before turning informer, told investigators that when he became a Lucchese made man in 1982, he was told that Gigante was the boss of the Genovese family. Ever since the death of boss Vito Genovese in 1969, the real family leader had been Philip "Benny Squint" Lombardo. Over the years, Lombardo used several front bosses to hide his real status from law enforcement, a practice continued when Gigante took over the family upon Lombardo's retirement in 1981. When Gigante took over the family, he reduced Salerno to a mere soldier. However, Salerno soon became bored and wanted to return to his rackets, and Gigante agreed.
On February 25, 1985, Salerno and eight other New York bosses on the "Mafia Commission" were indicted in the Mafia Commission Trial. The trial started in September 1986 and lasted three months. In October 1986, Fortune Magazine named the 75-year-old Salerno as America's top gangster in power, wealth and influence. For that reason, he was nominally the lead defendant in the trial. Many observers disputed Salerno's top ranking, claiming that law enforcement greatly exaggerated Salerno's importance to bring attention to their legal case against him. Salerno's bail request was denied and his attorneys appealed the decision all the way to the Supreme Court. However, in United States v. Salerno the Supreme Court ruled that he could be held without bail because of his potential danger to the community. On November 19, 1986, Salerno was convicted on RICO charges. The following January, he was sentenced, along with six other defendants, to 100 years in prison.
While awaiting the Mafia Commission trial, Salerno was indicted on March 21, 1986 in a second federal racketeering indictment. The indictment accused Salerno of infiltrating concrete companies to control the construction of Mount Sinai School of Medicine, Memorial Sloan-Kettering Cancer Center, and other Manhattan high-rise projects. Salerno was also accused of illegally aiding the election of Roy Lee Williams to the national presidency of the Teamsters Union. Salerno pleaded not guilty on all charges. He was convicted in 1988 and sentenced to 70 years in prison.
In 1986, shortly after Salerno's conviction in the Commission Trial, Salerno's longtime right-hand man, Vincent "The Fish" Cafaro, told the FBI that Salerno was only a front for Gigante. Cafaro, who had by this time turned informer, also revealed that the Genovese family had been keeping up this ruse since 1969. However, investigators had missed several clues that Salerno wasn't the real boss before then. Most tellingly, an FBI bug captured a conversation in which Salerno and capo Matthew "Matty the Horse" Ianniello were reviewing a list of prospective candidates to be made in another family. Frustrated that the nicknames of the wannabes hadn't been included, Salerno shrugged and said, "I'll leave this up to the boss"—a clear sign that he was not the real leader of the family. Nonetheless, this revelation would not have been enough to endanger Salerno's conviction and sentence in the Commission Trial; he had been convicted of specific crimes and not merely of being a boss. In any event, his 70-year sentence in the separate racketeering trial would have assured that he would die in prison.
After his conviction and imprisonment, Salerno's health deteriorated because of his diabetes and suspected prostate cancer. In July 1992, Anthony Salerno died of a stroke at the Medical Center for Federal Prisoners in Springfield, Missouri. He was buried at Saint Raymond's Cemetery in the Throggs Neck section of the Bronx in New York City.
In the 2011 gangster movie Kill the Irishman, Salerno was portrayed by Paul Sorvino. The film depicts Salerno's role in the gangland war between the Cleveland crime family and Irish mob boss Danny Greene.
Salerno helped inspire The Simpsons character, Fat Tony.