The New York Mets drafted Allen out of Bishop Ward High School in Kansas City, Kansas, in the eleventh round of the 1976 Major League Baseball draft. He went 10-2 with a 2.79 earned run average and led the Carolina League with 126 strikeouts with the Lynchburg Mets in his second professional season.
Allen came up with the Mets as a starting pitcher in 1979, and he made his major league debut on April 15 against the Philadelphia Phillies and former Met Nino Espinosa, giving up three runs in six innings and taking the loss. Allen was 0-5 as a starter when the Mets moved him to the bullpen. He won his next four decisions in a row as a reliever, and on July 28, he earned his first major league save.
Soon Allen emerged as the club's closer, earning eight saves by the end of the season and 69 total in his Mets career. In May of 1981, the Mets reached a deal to acquire Ellis Valentine from the Montreal Expos for Dan Norman and either Allen or Jeff Reardon. Unwilling to part with their closer, the Mets sent Reardon to the Expos.
The Mets converted Allen back to a starter in 1983, and on June 15, he and Rick Ownbey were traded to the St. Louis Cardinals for Keith Hernandez. Allen went 10-6 mostly as a starter his first season in St. Louis. In 1984, he was returned to the bullpen making only one emergency start.
By 1985, the Mets and Cardinals had become archrivals atop the National League East. Hernandez emerged as a leader of his new team, and had become a fan favorite in New York City. Allen, meanwhile, was 1-4 with a 5.59 earned run average, and began incurring the wrath of Cardinals fans. On July 16, he was sold to the New York Yankees. Allen was 1-0 with one save and a 2.76 ERA out of the Yankees' bullpen.
Following the '85 season, Allen was traded to the Chicago White Sox with Scott Bradley and Glen Braxton for Ron Hassey, Matt Winters, Chris Alvarez and Eric Schmidt. The ChiSox converted him back to a starter, and he earned his first win of the season against Ron Guidry at Yankee Stadium on May 15. Allen gave up only one earned run, four hits and two walks in seven innings for the first White Sox victory over Guidry at Yankee Stadium since August 13, 1980.
His next start at Yankee Stadium was even better. On July 20, Allen pitched a complete game two hitter to lead the White Sox to an 8-0 victory over the Yankees. For the season, Allen went 7-2 with a 3.82 ERA.
The White Sox released Allen during the 1987 season after he posted an 0-7 record and 7.07 ERA. He signed with the Yankees for the remainder of the season, and returned again for 1988. He signed a minor league deal with the Cleveland Indians in 1989, making three appearances for the big league club. He pitched for the Cincinnati Reds' AAA affiliate, the Nashville Sounds in 1990 before retiring.
Allen began coaching shortly after his retirement. After a brief stint in the independent Texas–Louisiana League as pitching coach with the Mobile BaySharks, Allen joined the Toronto Blue Jays in 1996 as pitching coach of their New York–Penn League affiliate, the St. Catharines Stompers.
After four seasons with the Jays' organization, Allen returned to the Yankees in 2000 as pitching coach of the Staten Island Yankees. He was pitching coach for the Columbus Clippers from 2003 to 2004, and returned to that position again in 2006 after serving as the bullpen pitching coach for the New York Yankees during the 2005 season. Allen, while the Columbus Clippers pitching coach, introduced Chien-Ming Wang's sinker, which became his signature pitch.
In 2007, he joined the Rays' franchise, working in their minor league system with his final stop as the pitching coach for the Durham Bulls.
In November 2014, the Minnesota Twins hired him as their major league pitching coach.
On May 26, 2016, Allen was booked into Hennepin County jail on suspicion of DUI and suspended indefinitely by the Twins. After he completed a five-week outpatient treatment program at the Hazelden Betty Ford Foundation that the team had checked him in to, the Twins brought him back in early July.