Before he joined the Kinks, Avory was asked twice to rehearse on drums at the Bricklayers Arms pub in London during late May/early June 1962 for a group of musicians who were later to become the Rolling Stones. It has been said that he also went on to play at their first show at The Marquee Club on 12 July 1962, yet Avory himself says "I think Tony Chapman did the gig at the Marquee. I didn't. I just rehearsed twice in the Bricklayers Arms in Soho."
Avory joined the Kinks in January 1964, after their previous drummer Micky Willet left the band. Avory was hired to replace him after their management saw an advertisement Avory had placed in the trade magazine Melody Maker. Despite his ability, early Kinks recordings (including hits such as "You Really Got Me") commonly did not feature Avory on drums; producer Shel Talmy hired more seasoned session drummers (most notably Clem Cattini and Bobby Graham) for studio work well into 1965, but with Avory often providing supporting percussion. The first single A-side Avory played on was Everybody's Gonna Be Happy, and went on to play on the vast majority of Kinks material.
Avory was always considered the quietest and most easy-going member of the Kinks lineup and was Ray Davies's best friend. However, his turbulent working relationship with guitarist Dave Davies resulted in many legendary onstage fights. In the most notorious (and widely mis-reported) incident, at the Capitol Theatre, Cardiff, South Wales, in 1965, Avory hit Davies with his drum pedal (not the cymbal stand, which, according to later interviews with Avory "would have decapitated him"), in reprisal for Davies kicking over his drum kit as revenge for a drunken fight the previous night in Taunton, apparently won by Mick. He then fled into hiding for days to avoid arrest for grievous bodily harm. On other occasions, fuming, he would hurl his drumsticks at Dave.
Ultimately, the relationship between Avory and the younger Davies brother deteriorated to the point where Avory left the band. By agreement with Ray Davies, he ceased performing and recording with the band in 1984, but accepted an invitation to manage Konk Studios, where the band and the Davies brothers record most of their records - a position he has held ever since.
Ray explained the situation:
The saddest day for me was when Mick left. Dave and Mick just couldn't get along. There were terrible fights, and I got to the point where I couldn't cope with it any more. Push came to shove, and to avoid an argument I couldn't face....we were doing a track called "Good Day" and I couldn't face having Mick and Dave in the studio, so I did it with a drum machine. Dave said he wanted to replace Mick, and [...] I took Mick out, and we got very, very drunk. We were in Guildford, and after about five pints of this wonderful scrumpy, Mick said if any other band offered him a tour, he wouldn't take it, because he didn't want to tour. And I remember him getting the train back - because he was banned from driving; it was a very bad year for Mick - and he walked to the station and disappeared into the mist.
Avory was replaced by Bob Henrit, former drummer with The Roulettes , Unit 4 + 2 and Argent.
Later on, it would seem that Dave Davies and Avory settled their differences, as Avory subsequently played the drums on "Rock 'n' Roll Cities", a track on the Think Visual album written by Dave Davies. Avory was asked to rejoin by Ray Davies, but he declined as he wanted a rest from the non-stop touring, working and performing schedule of two decades.
In 1996, he started playing with The Kast Off Kinks, along with John Dalton, Dave Clarke (no relation to the Dave Clark of "The Dave Clark Five"), and John Gosling. He has performed with them ever since.
In the 1990s, he also formed Shut Up Frank with Clarke, Noel Redding and Dave Rowberry of The Animals. They toured extensively and recorded several albums.
Avory was inducted into both the Rock and Roll Hall of Fame in 1990 and the UK Music Hall of Fame in 2005, with original bassist Pete Quaife and the Davies brothers.
By April 2004 at the request of The Animals, who were about to do their 40th anniversary tour, Chip Hawkes (formerly of The Tremeloes) was asked to form a band to tour along with them. This he did and brought together a true beat-era supergroup. The band features former original members of British 1960s groups, including Avory, Eric Haydock (The Hollies) and Hawkes, who have now combined to perform as The Class of 64 (referring to the actual year the British Invasion took America by storm), also featuring guitarists 'Telecaster Ted' Tomlin and Graham Pollock. The band have toured around the world, and have recorded an album of their former bands' hits.
In 2007, Avory left The Class of 64 and, with other former 64 members Haydock, Pollock, Tomlin, formed a new band called The Legends of the Sixties, which also features Martin Lyon. Avory made a special guest appearance onstage at Ray Davies' Royal Albert Hall performance on 10 May 2007. He played tambourine. Also guesting was Ian Gibbons, the former longtime Kinks keyboard player.
Avory also currently plays in The '60s All Stars' band, featuring members of British 1960s groups: John Dee (The Foundations), Alan Lovell (The Swinging Blue Jeans) and Derek Mandell (The George Harrison Band). The band can be seen playing regularly at the Cardinal Wolsey pub in Hampton Court in Avory's native East Molesey, Surrey. Avory was selected to pick up the sticks for From The Jam following Rick Buckler's departure, and toured with them in December 2009.
Avory lives in Kew, west London. Avory also has a daughter, although he has claimed that she wasn't "terribly musical."