Carroll was born in Evansville, Indiana. In 1992, he graduated from Castle High School in Newburgh, Indiana. He attended the University of Evansville where he led the Purple Aces to a four-year record of 137-95 and a runner-up finish in the Midwestern Collegiate Conference tourney. He was selected as an All-American infielder after his senior season (1996). Carroll's name appears 27 times in the Purple Aces' record book.
Drafted by the Montreal Expos in the 14th round of the 1996 Major League Baseball amateur draft, Carroll would make his Major League Baseball debut with the Expos on September 11, 2002 against the Chicago Cubs. He played third base and got two hits in three at-bats in that game. His first hit was a single to left field in the fourth inning against Alan Benes.
On October 3, 2004, Carroll scored the last ever run for the Expos franchise, as they were later relocated to Washington, D.C. Carroll was also the on-deck batter when Endy Chávez made the final out in Expos history at Shea Stadium.
On February 11, 2006, Carroll was traded to the Colorado Rockies for cash considerations.
He finished the 2006 season with a .300 batting average, 5 home runs, 36 RBI, and 10 stolen bases. Carroll also hit particularly well at Coors Field, finishing with a .375 clip in Denver compared to the .220 mark he amassed on the road. He played third base, shortstop, and second base, seeing by far the most action at second, where he appeared 109 times and made 102 starts. He committed just three errors as a second baseman, five overall. Carroll led all National League second basemen in fielding percentage.
On August 11, 2007, Carroll hit his first career grand slam as a pinch hitter against Chicago Cubs' pitcher Rich Hill in the sixth inning to break a 2–2 tie. The Rockies won that game, 15–2, as Carroll finished the game 1-for-2 with 2 runs and 5 runs batted in.
Carroll won the 2007 Wild card for the Rockies on October 1, 2007 with a sacrifice fly to right field, scoring Matt Holliday, to give the Rockies a 9–8 13-inning victory over the Padres.
In December 2007, Carroll was traded to the Cleveland Indians for minor league pitcher, Sean Smith.
On December 16, 2009, Carroll accepted a 2-year contract with the Los Angeles Dodgers. He saw extended action at shortstop in 2010 due to injuries to Rafael Furcal. He appeared in 133 games with the Dodgers, hitting .291. Due to continuing injury problems among the other infielders in 2011, Carroll appeared in a career high 146 games and hit .290. His 16 RBI on the season, tied with Dave Roberts and Tony Smith for the fewest ever by a Dodger with at least 400 plate appearances and put him in third place in Major League history in that category. He became a free agent after the season.
Carroll signed with a two-year, $6.75 million contract with the Minnesota Twins on November 15, 2011.
Carroll was thrown out of a game for the first time in his career on May 25, 2012. He was ejected by umpire Alan Porter after Porter called him out at first base. Later, Carroll claimed that, "Tie goes to the runner." On August 5, 2013, with the Twins trailing 13-0 in the 8th inning, Carroll became the 10th Twins position player to pitch in a game. He faced 3 Royal batters and retired all 3, throwing only 9 pitches, 7 of them strikes.
On August 11, 2013, he was traded to the Kansas City Royals for a player to be named later or cash. On October 11, 2013 the Royals outrighted him to the minor leagues, but he declined the assignment and became a free agent.
On January 9, 2014 he signed a minor league contract with the Washington Nationals. He was released on March 25.
On January 12, 2015, the Pittsburgh Pirates announced that Carroll would be joining their front office as a special assistant. This was cited by some sources as his official retirement as a player.
On February 28, 2008, Carroll's wife Kim gave birth to fraternal twins. His brother Wes Carroll is the head coach at Evansville. Jamey and Wes appeared in spring training games together with the Washington Nationals.
He resides in Rockledge, Florida.