Chambers' early career mirrored that of his twin brother Adam Chambers. Both played for school football teams - Grove Vale Primary School and Dartmouth High School, West Bromwich District and Sunday league team - Holy Name F.C. before joining West Bromwich Albion in 1996, and signing for West Bromwich Albion as apprentices in July 1997.
After progressing through Albion's youth system, both twins turned professional in January 1999. James made his debut in a 0–0 draw with Port Vale on 15 January 2000. He made 78 appearances in total for the Baggies, with a significant number coming in the 2000–01 season. In the 1998–99 season he and his brother Adam became the first twins to represent England at any level when they appeared together during the World Youth Tournament in Nigeria. They were also the first twins to play for the Albion, and the first to play in same side for any club in the Premier League.
With few chances available at the newly promoted Premiership side, he was allowed to move to Championship side Watford on loan in August 2004.
Chambers impressed manager Ray Lewington enough to sign him on a permanent three-year contract in a deal worth up to £250,000. Chambers started regularly in his first season at the club and also featured in the club's run to the League Cup semi-finals that year. He scored his only two career goals to date (as of December 2010) in the 4th round 5–2 victory over Premiership side Southampton. He also played in the quarter-final victory over Portsmouth and semi-final defeat to Liverpool.
Chambers competed with Lloyd Doyley for the right-back spot in 2005–06, as well as spending some time playing at left-back. Watford reached the play-offs, finishing third, and Chambers started on the right-wing in the semi-final first leg against Crystal Palace, putting in an impressive display. He kept his place for the second leg and the final, against Leeds United, where his deflected strike led to an own goal by Leeds goalkeeper Neil Sullivan. Watford won 3–0, and were promoted to the Premiership. In the post-match dressing room celebrations, Sky cameras inadvertently showed Chambers fully naked – exposing his private parts on national television – whilst jumping around and spraying manager Aidy Boothroyd with champagne.
Chambers started Watford's first game in the top-flight, a 2–1 loss to Everton where he was substituted at half-time. Aside from a League Cup appearance against Accrington Stanley, he did not play another first team match until he joined Cardiff on a month's loan in October 2006. Playing as a left back for the early Championship leaders, he started all seven games in his time at Ninian Park. Watford then rejected Cardiff's request to extend the loan.
He returned to the Watford side at right-back, but sustained an injury against Newcastle United that kept him out of the team until March. He featured sporadically after that, as Watford were relegated from the Premiership, but did play in the side's FA Cup quarter-final victory over Plymouth Argyle and semi-final loss to Manchester United.
In June 2007 Chambers turned down a new contract at Watford to sign a three-year contract with fellow Championship side Leicester City.
On 4 August 2008, Chambers signed a three-year-deal with Doncaster Rovers. Chambers had a clause in his contract at Leicester allowing him to move on a free transfer should the club be relegated. This clause was activated following the Foxes' relegation into League One. He became Rovers' seventh signing of the summer, leading up to their first season back in the second tier of English football after fifty years in the lower leagues.
Chambers was a regular for Rovers throughout the 2008–09 and 2009–10 seasons, with his ability to play on either side of the back four adding some much needed versatility to the Rovers ranks. Rehabilitation from a knee operation caused him to miss the beginning of the 2010–11 season.
After a period on trial at the club, Chambers joined League One side Walsall on 18 August 2012, again linking up with his brother Adam. Chambers emerged as a key part of the Saddlers squad over the following three seasons, making one hundred and eight appearances for Walsall and featuring in the 2015 Football League Trophy Final. He announced his retirement from football on 17 July 2015 to focus on his business interests.
In September 2016 he came out of retirement to sign for Worcester City.