Caldwell was born in Tampa, Florida in 1985. He attended Tampa's Jefferson High School, where he played football and ran track. In football, he played quarterback and also lined up at receiver and returned punts for the Dragons. His senior year, he moved from wide receiver to starting quarterback, and passed and rushed for over 2,220 yards and thirty-eight touchdowns while leading the Dragons to a Florida Class 4A state championship. He was the Tampa Tribune's Hillsborough County Player of the Year, an all-state selection, and was named a Parade magazine, SuperPrep and USA Today high school All-American following his senior season in 2002. He played in the 2003 U.S. Army All-American Bowl.
Considered a five-star recruit by Rivals.com, Caldwell was listed as the No. 1 wide receiver in the nation in 2003.
Also a standout athlete, Caldwell competed in track as sophomore and junior. At the 2002 Florida Class 3A Region 3 meet, he recorded a personal-best time of 10.39 seconds in the 100 meters to win region title. He got a PR of 21.71 seconds in the 200 meters. As a Florida collegiate, he joined the Gators track & field team following the conclusion of spring football practice. He finished 19th in 100m with time of 10.74 in the only meet of the season at the SEC Outdoor Championships. This was his first competition on the track in four years.
Caldwell accepted an athletic scholarship to attend the University of Florida in Gainesville, Florida, where he played for coach Ron Zook and coach Urban Meyer's Florida Gators football team from 2003 to 2007. In part because of his brother Reche's legacy as a Gator, and in part because of his friendship with Florida commitment and quarterback Chris Leak, Caldwell chose to play for the Gators.
As a true freshman seeing limited playing time, he had just nineteen receptions. During his sophomore season in 2004, Caldwell's production jumped to forty-three catches and three receiving touchdowns, as well as two rushes for sixty-six yards and one score. He received a medical redshirt for the 2005 season after fracturing his leg in an early-season Southeastern Conference (SEC) game against the Tennessee Volunteers. During his junior season in 2006, he emerged as the Gators' second leading wide receiver in catches (57), yards (577) and touchdowns (6), and also threw for a touchdown pass in the Gators 38–28 victory over the Arkansas Razorbacks in the 2006 SEC Championship Game. As a junior in 2006, he threw a seven-yard touchdown pass to help the Gators win their seventh SEC championship. Four weeks later, the Gators won their first BCS Championship Game and the second national championship in team history.
Just days after the 2007 BCS National Championship Game, Caldwell announced he would return to Florida for his senior season. As a senior team captain in 2007, Caldwell recorded a career-high 761 yards and seven touchdowns on fifty-six receptions, had twelve carries for fifty-eight rushing yards and a touchdown, and was the recipient of the Gators' Fergie Ferguson Award as the "senior football player who displays outstanding leadership, character and courage." Caldwell finished his senior season with 100-yard receiving games against the Western Kentucky Hilltoppers, Vanderbilt Commodores, South Carolina Gamecocks, and Florida Atlantic Owls. His 185 receptions in his four-year college career remains the best career total in Gators history.
Caldwell graduated from the University of Florida with a bachelor's degree in sociology in December 2007.
The Cincinnati Bengals chose Caldwell in the third round (ninety-seventh overall pick) in the 2008 NFL Draft, and played for the Bengals for four seasons from 2008 to 2011. In his rookie year, he made contributions in late season as a receiver and kickoff returner, and finished with eleven receptions for seventy-eight yards, a 26-yard average on thirteen kickoff returns, and five rushes for fifty-three yards during his rookie season. Caldwell caught his first career touchdown, a game-winning touchdown against the Steelers on September 27, 2009. In Week 5, Caldwell caught another game-winning touchdown pass against the Baltimore Ravens with 20 seconds to go in the game.
Caldwell signed a two-year deal with the Denver Broncos on March 21, 2012. He was re-signed by the Broncos to a two-year deal on March 11, 2014. The contract is worth up to $3.5 million. On February 7, 2016, Caldwell was part of the Broncos team that won Super Bowl 50. In the game, the Broncos defeated the Carolina Panthers by a score of 24–10. Caldwell had one catch for 22 yards in the Super Bowl. His catch occurred on the Broncos' opening drive and contributed to them getting into field goal range.
On May 4, 2016, the Detroit Lions signed Caldwell to a one-year contract. On August 22, 2016, Caldwell was released by the Lions.
Caldwell is the younger brother of Reche Caldwell, former NFL wide receiver and Florida Gators star.