Williams was born in Hereford and attended the nearby football academy at Holme Lacy, playing for the Hereford United youth team. He also gained experience by attending football academies at Aston Villa and Birmingham City.
Williams made his debut for the Hereford United first team in the 2003–04 season, coming on as a substitute in the Herefordshire Senior Cup final against Forest Green. The following season, he featured regularly on the bench and made three league appearances as a substitute. He also started the two GLS Conference Cup games against Burton and Northwich.
2005–06 was his breakthrough season. His appearances for the first team were initially from the bench but he was given his first start in the league against Woking, scoring the opening goal. He progressed rapidly and went on to become the club's top scorer for the season, scoring 13 goals from 45 appearances in all competitions. Most notably he scored two vital goals in the playoff semi-final against Morecambe, and in the final itself against Halifax. He signed a new contract with the club in March 2006.
His first season in the Football League saw him played out of position on the right wing for the majority of the season resulting in 48 first team appearance, with only Rob Purdie and Alan Connell making more appearances. Williams' eight goals also made him third highest goalscorer, and he was awarded the Away Goal of the Season for his goal against Bury.
Williams was the subject of two transfer bids from Bristol Rovers during the course of the season. At the end of August a £100,000 bid was rejected by Hereford, but at the end of January a £150,000 bid was accepted. However Williams turned down a move to the Gasheads, having already stated he was happy at Hereford and was not interested in moving to a club in the same division. When Bristol Rovers gained promotion to League One, speculation intensified that he would make the move, and on 5 July 2007 the transfer was finally confirmed.
Williams made his debut for Bristol Rovers on 11 August 2007, scoring four minutes after appearing as a substitute for Rickie Lambert.
On 1 September 2008, Williams rejoined former club Hereford United on loan until 5 January 2009. An injury hampered his return to Edgar Street but he scored his first goal in the win over Carlisle United on 21 October. On expiry of his loan he returned to Bristol Rovers, but just 24 hours later he was loaned back to Hereford for the remainder of the season.
He was placed on the transfer list at the end of the 2008–09 season, but after some competitive performances from the bench he was removed from the list early on during the start of the 2009–10 season. On 29 September 2009, in an away 3–2 victory against Southampton, he scored a left-footed shot from the edge of the area, this goal was later nominated for Goal of the Year 2009.
In May 2010, Williams rejected a new deal with Bristol Rovers and signed a two-year contract with fellow League One side and local rivals Yeovil Town, he linked up with them on 1 July 2010. On 7 August 2010, Williams made his debut for Yeovil against Leyton Orient, and scored his first goal for the club against Rushden & Diamonds in the FA Cup, and scored his first league goal in a 3–1 defeat against Dagenham & Redbridge the following week. Williams first season with Yeovil saw him score eight goals in 39 matches mostly operating from the wing.
He started the new season well with a goal in a 3–1 win at home to Oldham Athletic before being out injured for over a month. However, once he had returned from injury he scored again, this time in a 3–0 win away at Hereford United in the FA Cup. Williams ended the campaign as Yeovil's top scorer with 17 goals in all competitions but rejected a new contract to remain at the club.
Having rejected Yeovil's offer of a new contract, on 18 June 2012, it was confirmed Williams had signed for newly promoted League One side Swindon Town on a two-year contract.
On 23 August 2013, Williams rejoined Yeovil Town on a season long loan deal, only 14 months after previously leaving for Swindon Town. Williams made his second debut for Yeovil as a second-half substitute against Derby County, on 24 August 2013. Williams made a total of nine appearances without scoring before picking up serious cruciate injury in training ruling him out for the rest of the season. Despite being ruled out with injury for the rest of the season, due to the loan agreement with Swindon Town Williams remained officially on loan at Yeovil for the remainder of the season, with Yeovil still having to pay a portion of his wages.
On 19 June 2015, Williams signed for Doncaster Rovers on a three-year deal, despite interest from several other clubs including Bradford City, Scunthorpe United, Sheffield United and Millwall. On the 27th of August Andy scored his first professional hat-trick in a 4–1 win vs Yeovil Town F.C As of 10 January 2017.