Richardson began his career with Leeds United where he spent nine years making 186 appearances only interrupted by two loan spells at Stoke City in 2003. He joined Charlton Athletic in July 2009 where he spent the 2009–10 season. He then had spells with Southampton, Middlesbrough and Ipswich Town before joining his home-town club Rotherham United in June 2014.
Richardson made his Leeds debut as a second-half substitute against Hapoel Tel Aviv during the 2002–03 UEFA Cup campaign. He made his full debut against Arsenal at right full-back, a position he was tipped to have on a permanent basis after the retirement of first team regular Gary Kelly. Richardson suffered Premiership relegation with Leeds in 2004. Since then, Richardson covered a variety of roles for Leeds. He spent two spells on loan at Stoke City in 2003, scoring once against West Ham.
Richardson scored his first Leeds goal on his second start for the club, the goal turning out to be the winner against Derby County in the opening game of the 2004–05 campaign (because the game was a 12:15 kick-off on Sky Sports, this was actually the first ever goal in the newly rebranded "Football League Championship"). Richardson was then replaced on the right of midfield by John Oster. However, an injury to Stephen Crainey meant Richardson was given a chance to establish himself at left full-back. However, his spell was short lived after Manager Kevin Blackwell chose to play the likes of Danny Pugh and Matthew Kilgallon in that position.
Prior to the 2005–06 campaign, Richardson was subject to two bids from Sunderland, both which were rejected by Leeds' chairman, Ken Bates. Several months later, Richardson signed a new contract with Leeds, pledging his future to the club until August 2008.
That season, Richardson infrequently played on the right of midfield – competing for his position with Ian Moore (himself preferring to play as a forward), Jonathan Douglas and David Healy (who preferred to play out of position rather than as a substitute). Richardson scored in the Play-off semi final away leg win against Preston North End for Leeds in May 2006, playing in an unusual attacking wide right position, though he reverted to a more familiar role for the Play-off final defeat against Watford a few weeks later. In the 2006–07 season, Richardson hardly featured under Blackwell in the Leeds team at all due to injuries. Under new manager Dennis Wise, Richardson played more regularly in the Leeds starting line-up, replacing the injured Gary Kelly at right back.
On the retirement of Kelly, Richardson was given the number two shirt and became the club's first choice right back and started all the club's matches in the 2007–08 season until injury ruled him out on 2 February. Under former Leeds manager Gary McAllister, Richardson also played in the left-back position, playing a few games there as a replacement for on-loan Leicester City defender Alan Sheehan. After a poor performance in this position against Cheltenham Town, Richardson was booed off the pitch by Leeds fans and did not play for a few weeks after. He returned in his usual right-back position as a replacement for the injured Darren Kenton in their 1–0 win over Doncaster Rovers.
Richardson was appointed Leeds captain in August 2008 by McAllister, and remained club captain despite injury under new manager Simon Grayson. Richardson made his final appearance for Leeds as a late substitute against Northampton Town, this was to be his first game for Leeds for several weeks, and also it proved to be his last appearance for the Elland Road side.
Leeds United confirmed on 16 June 2009 that they would not continue contract negotiations with him due to the signing of former Northampton captain Jason Crowe. Richardson was released due to his contract ending on 30 June 2009. In July 2010, Lloyd Sam revealed that Richardson had never wanted to leave Leeds.
On 9 July, Richardson signed for Charlton Athletic on a free transfer on a two-year deal. He scored his first for the club in a 2–2 draw with Gillingham on 20 March 2010. In May 2010, he was selected in the Football League One PFA Team of the Year for the 2009–10 season. Richardson was part of the Charlton side which reached the League 1 playoffs, they were knocked out of the playoffs after losing on penalties to Swindon Town, Richardson scored his penalty, but Charlton lost and Richardson suffered his fourth playoff campaign defeat in his career.
On 6 July 2010, Richardson signed for League One club Southampton for an undisclosed fee, in a deal reportedly worth £450,000. After the club's second pre-season game, against FC Sochaux-Montbéliard on 17 July 2010, the club revealed that Richardson had suffered an injury in the form of a dislocated shoulder "during a routine training session as he went to make a headed clearance and landed awkwardly". The new signing was flown back to England immediately for treatment, and the club estimate that he will be out of action for approximately three months, ruling him out of the beginning of the season. He made his debut in a 3–1 victory away at Notts County. Richardson began the 2011–12 season as first choice right back for Southampton, featuring in the first game of the season against former club Leeds United, where Southampton were 3–1 winners.
On 4 June 2013 he was released by the club.
On 2 August 2013, Richardson was signed by Middlesbrough. He made his full debut on 10 August in a 1–0 win over Charlton Athletic at The Valley. Richardson joined Ipswich Town on loan until the end of the 2013–14 season on 25 January 2014.
Richardson joined his home-town club Rotherham United on 27 June 2014, signing a two-year contract. He spent two seasons with the Millers before joining fellow South Yorkshire side Doncaster Rovers in the summer of 2016. His time at the Keepmoat Stadium was disrupted by injury and he left the club in December 2016.As of 16 February 2016
Football League One runners-up: 2010–11
Football League Championship runners-up: 2011–12