Lumley was born at Tickhill Castle, the second son of Richard Lumley, 9th Earl of Scarbrough and Frederica Drummond, granddaughter of the fifth Duke of Rutland. On his paternal grandmother's side of the family, he was descended from the Beresford family; his notable Irish relatives included Bishop George Beresford and the Earl of Tyrone. He was educated at Eton. His elder brother Lyulph, Viscount Lumley died in 1868 at age 18, leaving Aldred as heir to the family titles.
In 1877, he was gazetted to the 7th Hussars. He served for six years and saw service in the Anglo-Zulu War in 1879. He was in Natal at the outbreak of the First Boer War in 1881. In 1883, he left the service to assist his ailing father in the managing of their estates at Sandbeck Park and Lumley Castle. After his father's death the following year, he was forced to let the family seat of Sandbeck Park due to the Great Agricultural Depression.
Scarbrough was an avid sailor and member of the Royal Yacht Club. For six years, he sailed around the world, visiting India, Africa, the West Indies, and Central and South America. He travelled with Frank Linsly James aboard the HMS Waterwitch and was with him when James was killed by an elephant in 1890 in Gabon.
He was a member of the council of the Royal Niger Company and during this time visited Africa with Sir George Goldie to make treaties with tribal chiefs. During his travels, he developed a keen interest in botany.
He was Lord Lieutenant of the West Riding of Yorkshire from 1892 to 1904. In early February 1900 he was appointed second in command of a Battalion of Imperial Yeomanry, with the temporary rank of Major in the Army. His battalion left for South Africa to serve in the Second Boer War, and later the same year he commanded the 7th Hussars, the Yorkshire Dragoons and the Yorkshire Mounted Brigade during the war. After his return to the United Kingdom, he was appointed an aide-de-camp to King Edward VII in the 1902 Coronation Honours list on 26 June 1902, with the regular rank of colonel. He served as such until the King's death in 1910. He was Director-General of the Territorial Force until his retirement in 1921.
Scarbrough was appointed Knight Grand Cross of the Order of the British Empire (GBE) shortly before his retirement from the Territorial Force in 1921 for his military service. He was knighted and was a Sub-Prior of the Order of the Hospital of Saint John of Jerusalem from 1923 to 1943.
In addition to his estates in Yorkshire, the earl owned considerable land around the seaside town of Skegness, Lincolnshire, which became accessible by railway in 1873. At that time it was a small fishing village. Recognising its potential value as a holiday destination, Scarbrough and his business agent planned to transformed Skegness into a resort town. For three decades, he helped spur the town's growth with his plans, included constructing a large pier, a church, tree-lined promenades, parks, gardens, houses and hotels. A village of just 500 people in 1850, the town grew to 2,000 permanent residents by the turn of the century.
On 8 April 1899, Scarbrough married Lucy Cecilia Dunn-Gardner (d. 24 November 1931), widow of Col. Robert Ashton (d. 1898), at Christ Church, Mayfair. Lucy was the daughter of Cecil Dunn-Gardner and was herself made a Dame of Grace of the Order of Saint John of Jerusalem for her work with the hospital. His wife brought a stepson and stepdaughter to their marriage. She and the earl had a single daughter, Lady Serena Lumley (1901–2000), who married the Hon. Robert James, third son of Baron Northbourne. Their elder daughter Ursula James married David Allan Bethell, 5th Baron Westbury (1922–2001) in 1947; the princesses Elizabeth and Margaret and the Duke and Duchess of Gloucester attended their wedding at St Martin-in-the-Fields. Their son Richard is the present baron.
Scarbrough's stepdaughter Dorothy Violet Ashton (1885–1956) married in 1914 Lord Gerald Wellesley (1885–1972), who succeeded his nephew as Duke in 1943. The marriage was not a great success, but produced a son and daughter. The son was the 8th Duke of Wellington (1915–2014). Lady Serena later refused the marriage proposal of her brother-in-law, by then Duke of Wellington, after their respective spouses had died.
The earl died in 1945 and was succeeded in the family titles by his nephew, Sir Roger Lumley.