It is believed that Zwillman was born on July 27, 1904, in Newark, New Jersey. He was forced to quit school in order to support his family after his father's death in 1918. Zwillman first began working at a Prince Street cafe, the headquarters of a local alderman in Newark's Third Ward. However, in need of more money, Zwillman was eventually forced to quit, later selling fruits and vegetables in his neighborhood with a rented horse and wagon.
Zwillman was unable to compete with the cheaper Prince Street pushcarts, however, so he moved to the more upper-class neighborhood of Clinton Hill, where he began selling lottery tickets to local housewives. He observed that much more money was made selling lottery tickets than produce, so he concentrated on selling lottery tickets through local merchants. By 1920, Zwillman controlled the bulk of the numbers racket with the help of hired muscle.
At the start of Prohibition, Zwillman began smuggling whiskey into New Jersey through Canada, using several World War I armored trucks. Zwillman later joined a syndicate headed by Joseph Reinfeld to smuggle liquor from Canada using ships. Zwillman used this revenue to greatly expand his operations in illegal gambling, prostitution, and labor racketeering, as well as legitimate businesses, including several prominent night clubs and restaurants.
Zwillman dated actress Jean Harlow at one time and got her a two-picture deal at Columbia Pictures by giving its head, Harry Cohn, a huge loan. He also bought Harlow a jeweled bracelet and a red Cadillac. He referred to her in derogatory terms to other mobsters in secret surveillance tapes. He later married Mary Mendels, the only daughter of Eugene Mendels—a founder of the American Stock Exchange (then known as the Curb Exchange).
After Dutch Schultz's murder in 1935, Zwillman took over those of Schultz's criminal operations that were in New Jersey. The press began calling Zwillman the "Al Capone of New Jersey." Zwillman was particularly close to Capone and was his partner in controlling the movie projectionist union. In the mid-1940s, he went into hiding while the union leaders were tried for forcing Hollywood studios to pay huge bribes to ensure that their theaters would remain open. He and Capone were reportedly paid millions of dollars annually.
However, Zwillman often sought to legitimize his image, offering a reward for the return of the Lindbergh baby in 1932, and contributed to charities, including $250,000 to a Newark slum-clearing project.
Shortly after taking over Schultz's operations, Zwillman became involved in local politics, eventually controlling the majority of local politicians in Newark for over twenty years. During the 1940s Zwillman, along with long-time associate Willie Moretti, dominated gambling operations in New Jersey, in particular the Marine Room inside Zwillman's Riviera nightclub, The Palisades.
In 1951, Zwillman's activities were a major focus by the Kefauver Committee on organized crime. While Zwillman acknowledged that he was a bootlegger during Prohibition, he insisted that his subsequent businesses were legitimate.
Zwillman was also close to many celebrities, including Joe DiMaggio. When Zwillman was being investigated by the Kefauver Committee along with other alleged "Outfit" members he reportedly planted three trunks full of money with DiMaggio to hide it from the IRS. It was not returned after Zwillman's death.
In 1956, Zwillman was tried for income tax evasion. However, the jury became deadlocked on the charges and the jury was dismissed. However, several associates of Zwillman were subsequently arrested and charged with bribing two of the jurors.
During the 1959 McClellan Senate Committee hearings on organized crime, Zwillman was issued a subpoena to testify before the committee. However, shortly before he was to appear, Zwillman was found hanged in his West Orange, New Jersey, residence on February 27, 1959.
Although Zwillman's death was ruled a suicide, police found bruises on Zwillman's wrists, supporting the theory that Zwillman had been tied up before being hanged.
While his death was ruled a suicide because of Zwillman's intractable income tax problems, it is often speculated that Vito Genovese had ordered Zwillman killed. Others have alleged that Meyer Lansky, suspecting that the New Jersey gangster had agreed to become a government informant, gave permission for the Italian Mafia to take action against Zwillman. The theory that he was hanged was also supported by deported mobster Charles "Lucky" Luciano, who allegedly told journalist Martin Gosch in Italy that the suicide theory was nonsense, and that before hanging him, Zwillman's killers had trussed him up like a pig. Martin Gosch's biography (which he co-authored with Richard Hammer) of Lucky Luciano is somewhat controversial and considered fictional by many mob experts. However, the authors have claimed that the contents are entirely based on interviews with Luciano, who died before the book was ever published.