Freeman-Mitford was the son of Henry Reveley Mitford (1804–1883) of Exbury House, Exbury, Hampshire and the great-grandson of the historian William Mitford, and was educated at Eton and Christ Church, Oxford. While his paternal ancestors were landed gentry, whose holdings had once included Mitford Castle in Northumberland, his mother (Georgiana) Jemima was a daughter of the courtier the 3rd Earl of Ashburnham, with a noble ancestry through the earls of Beverley. His parents separated in 1840 when Redesdale was just three years old, and his mother remarried to a Mr. Molyneaux.
Like his cousin Swinburne, he was named Algernon after his grandfather Algernon Percy, 1st Earl of Beverley.
Entering the Foreign Office in 1858, Mitford was appointed Third Secretary of the British Embassy in St Petersburg. After service in the Diplomatic Corps in Shanghai, he went to Japan as second secretary to the British Legation at the time of the migration of the Japanese Seat of Power from Kyoto to Edo (modern-day Tokyo, known as the "Meiji Restoration". Mitford`s memoirs recount the troubled time of the foreign settlements at Kobe over the fortnight following American Rear-Admiral Henry Bell`s death, and the death of British consul Francis Gerard Mijburgh. Rededale served as secretary under Myburgh`s replacement, John Frederik Lowder. There he met Ernest Satow and wrote Tales of Old Japan (1871), a book credited with making such Japanese Classics as "The Forty-seven Ronin" first known to a wide Western public. He resigned from the diplomatic service in 1873.
Following the 1902 Anglo-Japanese Alliance, in 1906 he accompanied Prince Arthur on a visit to Japan to present the Emperor Meiji with the Order of the Garter. He was asked by courtiers there about Japanese ceremonies that had disappeared since 1868. He is one of the people credited with introducing Japanese knotweed to England.
From 1874 to 1886, Mitford acted as secretary to HM Office of Works, involved in the lengthy restoration of the Tower of London and in landscaping parts of Hyde Park such as "The Dell". From 1887, he was a member of the Royal Commission on Civil Services. He also sat as Member of Parliament for Stratford-on-Avon between 1892 and 1895.
According to W. S. Gilbert, Mitford served as a consultant on Japanese culture to Gilbert and Arthur Sullivan during the development of their 1885 Savoy Opera The Mikado. A traditional Japanese song hummed by Mitford to Gilbert and Sullivan during a rehearsal was used in the opera for the march accompanying the Mikado's entrance.
In 1886, Mitford inherited the substantial country estates of his first cousin twice removed, John Freeman-Mitford, 1st Earl of Redesdale. In accordance with the will he assumed by Royal licence the additional surname of Freeman. Appointed a Deputy Lieutenant for Gloucestershire, he became a magistrate and took up farming and horse breeding. He was a member of the Royal Yacht Squadron from 1889 to 1914. Redesdale was President of the Royal Photographic Society between 1910–1912.
He substantially rebuilt Batsford House beside Batsford in Gloucestershire in the Victorian Gothic manorial style, but at such a cost that it had to be sold within a few years of his death. It was bought by Lord Dulverton and is still owned by his descendants.
In the 1902 Coronation Honours list it was announced that he would receive a barony, and the Redesdale title was revived when he was raised to the peerage as Baron Redesdale, of Redesdale in the County of Northumberland, on 15 July 1902. He took the oath and his seat in the House of Lords a week later, on 24 July.
During his time in Japan, he was said to have fathered two children with a geisha. Later, he was considered to be one of the possible fathers of Clementine Hozier (1885–1977), in the course of an affair with his wife's sister Blanche. Clementine married Winston Churchill in 1908.
In his closing years, Lord Redesdale edited and wrote extensive and effusive introductions for two of Houston Stewart Chamberlain's books, Foundations of the Nineteenth Century and Immanuel Kant: A Study and Comparison with Goethe, Leonardo da Vinci, Bruno, Plato, and Descartes, both two volumes each, translated into English by John Lees, M.A., D.Litt., and published by John Lane at the Bodley Head, London, in 1910 and 1914 respectively.
Lord Redesdale married in 1874 Lady Clementina Gertrude Helen (1854–1932), the daughter of David Ogilvy, 10th Earl of Airlie by his spouse Blanche, the daughter of Edward Stanley, 2nd Baron Stanley of Alderley. They had five sons and four daughters:Hon. Frances Mitford (born 1875), who later married Alexander Kearsey
Hon. Clement Freeman-Mitford (1876–1915), the eldest son, killed in action in the Battle of Loos, whose posthumous daughter Clementine married Sir Alfred Beit
Hon. David Freeman-Mitford (1878–1958), who succeeded his father in the barony and was the father of the prominent Mitford sisters
Hon. Iris Freeman-Mitford (1878–1966)
Hon. Bertram Freeman-Mitford (1880–1962), who succeeded David as the 3rd Baron Redesdale in 1958
Hon. John Freeman-Mitford (1885–1963), who succeeded Bertram as the 4th Baron Redesdale in 1962
Hon. Joan Freeman-Mitford (born 1887), who married Dennis Hebert Farrer in 1907
Hon. Ernest Freeman-Mitford (1895–1939), who was the father of Clement Freeman-Mitford, 5th Baron Redesdale
Hon. Daphne Freeman-Mitford (1895–1996), who married George Bowyer, 1st Baron Denham