Jaffee, who was Jewish, was born to Jewish parents who had emigrated from Russia in 1896. He grew up in the Crotona Park section of The Bronx, where he played baseball with future Hall of Famer Hank Greenberg. He briefly attended DeWitt Clinton High School, but dropped out after failing to make the varsity baseball team.
At age 14, Jaffee took up skating at the Gay Blades of Iceland rink (which later became the Roseland Ballroom). Rather than pay the 75-cent admission fee, he worked as an ice cleaner to gain admission. He entered numerous skating races in the 1920s. He finally won the Silver Skates two-mile race in 1926, won the national five- mile event the following year, and qualified for the U.S. Olympic team in 1928.
At the 1928 Winter Olympics in St. Moritz, Jaffee finished fourth in the 5000-meter skate, the best finish by an American in that event to that date. In the later 10,000-meter race, Jaffee was leading the competition, having outskated Norwegian defending world champion Bernt Evensen in their heat, when rising temperatures thawed the ice. In a controversial ruling, the Norwegian referee canceled the entire competition. Although the International Olympic Committee reversed the referee's decision and awarded Jaffee the gold medal, the International Skating Union later overruled the IOC and restored the ruling. Evensen, for his part, publicly said that Jaffee should be awarded the gold medal, but that never happened.
That year he also set a world record in the mile (2:30.6).
Jaffee competed again at the 1932 Winter Olympics in Lake Placid, New York. At the time, Jaffee recalled, there were signs in Lake Placid that said "No dogs or Jews allowed". There, he won gold medals in both the 5,000 and 10,000-meter races. In the 10,000-meter race, Jaffee won in a thrilling finish by leaping across the finish line ahead of Frank Stack and Ivar Ballangrud.
Jaffee served on the American board for the Second Maccabiah, along with Benny Leonard and Nat Holman.
During the Great Depression, the unemployed Jaffee ended up on bread lines and was forced to pawn his Olympic and other medals for $3500. He obtained a job on Wall Street, he tried to redeem his medals, but the pawn shop went out of business, meaning that he never saw the medals again.
In 1934, he worked as Winter Sports Director at Grossinger's Catskill Resort Hotel, and set a world record there by skating 25 miles in 1:26:01. breaking the 30-year-old record by five minutes.
Jaffee appeared in a full-page ad for Camel cigarettes in 1934, entitled "It Takes Healthy Nerves for Jaffee to be the World's Champion Skater; Steady Smokers Turn to Camels".
Jaffee was elected to the United States Skating Hall of Fame in 1940 and the International Jewish Sports Hall of Fame in 1979. He died in San Diego in 1981.