Allen was born in Fayetteville, North Carolina. He attended Terry Sanford High School in Fayetteville, and played high school football for the Terry Sanford Bulldogs. During his career, he had 68 receptions for 1,257 yards.
Allen attended Clemson University, where he played for the Clemson Tigers football team from 2008 to 2011. He was redshirted in 2008. In 2009, he started six of 14 games and had 10 receptions for 108 yards and three touchdowns. He started all 13 games in 2010 and was a second-team All-ACC selection after recording 33 receptions for 373 yards with a touchdown.
Through the first seven games of his junior season, Allen had 27 receptions for 381 yards and four touchdowns.
Allen was named the 2011 recipient of the John Mackey Award on December 8, 2011 at the Home Depot College Football Awards Red Carpet Show. He graduated from Clemson on August 9, 2014.
Allen was drafted 64th overall by the Indianapolis Colts in the 2012 NFL Draft. He was the second tight-end taken in the draft as well as the second taken by the Colts. He helped fellow rookie Andrew Luck lead the Colts to an 11-5 record. Despite their draft positions, Allen actually bested his teammate (taken 34th overall), Coby Fleener, in overall production ranking 7th best among the league's rookie receiving leaders. Allen and fellow rookie teammates T. Y. Hilton, Fleener, Vick Ballard, and LaVon Brazill combined for an NFL record 3,108 yards - the most by any rookie class playing for an NFL club since the 1970 NFL merger. Interestingly, the second best performance was the Colts 1999 rookie class with 2,751 yards. Allen finished the 2012 season with 521 receiving yards on 45 catches, and 3 touchdowns. Allen was intended to be a large part of new Colts offensive coordinator Pep Hamilton's offense in 2013, but was placed on the injured reserve following a hip injury in the first game of the season. He finished the season with one catch for 20 yards and a touchdown.
Allen battled a knee injury late in the 2014 season, but started 13 games for the Colts and finished with 29 receptions, 395 yards, and 8 touchdowns. On November 8, 2015, Allen was poked in the eye by Denver Broncos' cornerback Aqib Talib. Talib was given a one-game suspension for the incident the following day. On December 30, 2015, Allen was placed on injured reserve.
On March 7, 2016, Allen signed a four-year, $29.4 million extension with the Colts.
On March 9, 2017, the Colts traded Allen and a 2017 sixth-round draft pick to the New England Patriots for a 2017 fourth-round draft pick. The trade was the first between the Colts and Patriots since 1985.