Cavendish was born to Edward Cavendish, 10th Duke of Devonshire and Mary Cavendish, Duchess of Devonshire, the former Mary Alice Gascoyne-Cecil, daughter of James Gascoyne-Cecil, 4th Marquess of Salisbury. He was educated at Eton College and Trinity College, Cambridge.
Cavendish served in the British Army during World War II. Having attended an Officer Cadet Training Unit, he was commissioned into the Coldstream Guards as a second lieutenant on 2 November 1940. On 7 December 1944, while holding the rank of acting captain, he was awarded the Military Cross 'in recognition of gallant and distinguished services in Italy'. The action took place on 27 July 1944 when his company was cut off for 36 hours in heavy combat near Strada, Italy. He held the rank of major at the end of the war.
In later life, he took on a number of honorary positions within the military. On 2 December 1953, he was appointed Honorary Colonel of a Territorial Army unit of the Royal Regiment of Artillery. On 2 October 1981, he was appointed Honorary Colonel of the Manchester and Salford Universities Officers' Training Corps. He relinquished this appointment on 2 January 1985.
Cavendish ran unsuccessfully as a National Liberal candidate for Chesterfield in the 1945 general election and as a Conservative for the same seat in 1950. He was Mayor of Buxton from 1952 to 1954. He served as Parliamentary Under-Secretary for Commonwealth Relations from 1960 to 1962, Minister of State at the Commonwealth Relations Office from 1962 to 1963, and for Colonial Affairs from 1963 to 1964. He once said that these appointments by his uncle, Harold Macmillan, the then-prime minister, were "the greatest act of nepotism ever".
He joined the Social Democratic Party shortly after its foundation in 1981, but left the party when David Owen resigned as the party's leader in 1987, describing Owen as "the best of them". He then sat as a crossbencher during his rare appearances in the House of Lords.
The duke followed the family tradition of owning racehorses, the most famous of which was Park Top, the subject of the duke's first published book, A Romance of The Turf: Park Top, which was published in 1976. His autobiography, Accidents of Fortune, was published just before his death in 2004. The duke had many disputes over the years with the ramblers who used the paths near Chatsworth. Eventually though, in 1991, he signed an agreement with the Peak National Park Authority opening 1,300 acres (5 km²) of his estate to walkers. He said that everyone was "welcome in my back garden". The duke's real estate holdings were vast. In addition to Chatsworth he also owned Lismore Castle in Ireland and Bolton Abbey in North Yorkshire. He also owned the bookshop Heywood Hill and the gentleman's club Pratt's.
He was a major collector of contemporary British art, known especially for his patronage of Lucian Freud. He was one of the founders, and the chief patron of, the Next Century Foundation, in which capacity he hosted the private Chatsworth talks between representatives of the governments of the Arab World and Israel. The duke was listed at number 73 in the Sunday Times Sunday Times Rich List of the richest people in Great Britain in 2004.
In 1941 Cavendish married the Hon. Deborah Mitford (31 March 1920 – 24 September 2014), one of the Mitford sisters.
Three of the couple's six children died soon after birth, and the Duke's extramarital affairs became public after he appeared as a witness at a burglary trial and was forced to admit, under oath, that he was on holiday with one of a series of younger women when the crime occurred at his London home. The Duke, however, claimed that much of his marriage's success was due to the Duchess's tolerance and broadmindedness. The Duchess, as chatelaine, was largely responsible for the success of Chatsworth as a commercial endeavour.
Cavendish and his wife had six children, three of whom died in infancy. The three surviving children were a son, Peregrine Cavendish, 12th Duke of Devonshire, and two daughters, Lady Emma Cavendish and Lady Sophia Topley.Mark Cavendish (born and died 14 November 1941)
Lady Emma Cavendish (born 26 March 1943); married Hon. Tobias Tennant, son of Christopher Grey Tennant, 2nd Baron Glenconner on 3 September 1963. They have three children and ten grandchildren:
Isabella Tennant (born 28 June 1964) she married Piers Hill, son of Simon Hill, on 9 August 1997. They have three children:
Edward Tobias Tennant (born 30 March 1967); married Emma Bridgeman in 2000. They have three children:
Harry Tobias Tennant (born 7 September 2001)
Georgia Rose Tennant (born 9 July 2003)
Isla May Tennant (born 8 August 2006)
Stella Tennant (born 17 December 1970); married David Lasnet on 22 May 1999. They have four children.
Peregrine Cavendish, 12th Duke of Devonshire (born 27 April 1944); married Amanda Heywood-Lonsdale on 28 June 1967. They have three children and eight grandchildren.
Lord Victor Cavendish (born and died 22 May 1947)
Lady Mary Cavendish (born and died 5 April 1953)
Lady Sophia Louise Sydney Cavendish (born 18 March 1957); married Anthony Murphy on 20 October 1979 and they were divorced in 1987. She remarried Alastair Morrison, 3rd Baron Margadale on 19 July 1988 and they were divorced. They have two children. She remarried again William Topley on 25 November 1999.
In December 1946, the Duchess had a miscarriage; had the child been born, it would have been a twin of Victor Cavendish, born in 1947.
Cavendish's older brother William Cavendish, Marquess of Hartington, who would have inherited the dukedom, was killed in combat near the end of the war. With William's death, Andrew became heir and received the courtesy title of Marquess of Hartington, which he held from 1944 until 1950.
Cavendish's uncle, Lord Charles Cavendish, died aged 38 as a result of alcoholism. Lord Charles's will bequeathed Lismore Castle to Andrew upon the remarriage of Charles's wife, Adele Astaire, in 1947.
The 10th Duke died of a heart attack while visiting Eastbourne in November 1950 and Andrew, who was in Australia at the time, inherited the title. The Duke died while being attended by suspected serial killer Dr John Bodkin Adams, who was his doctor when visiting Eastbourne. No proper police investigation was ever conducted into the death but Cavendish later said "it should perhaps be noted that this doctor was not appointed to look after the health of my two younger sisters, who were then in their teens"; Adams had a reputation for grooming older patients to extract bequests.
Cavendish inherited the estate but also an inheritance tax bill of £7 million (£216 million in 2015), nearly 80 per cent of the value of the estate. To meet this, the Duke had to sell off many art objects and antiques, including several Rembrandts, Van Dycks and Raffaello Santis, as well as thousands of acres of land.
The Duke is buried in the churchyard of St Peter's Church, Edensor - in the grounds of Chatsworth.
In 1996 he was made a Knight of the Garter.
He once told an interviewer:
"Wonderful things have happened in my life — it's time my son had his turn. When I was young I used to like casinos, fast women and God knows what. Now my idea of Heaven, apart from being at Chatsworth, is to sit in the hall of Brooks's, having tea."