Born in Geneva, Switzerland, the fourth son of George Hay, 8th Marquess of Tweeddale and Lady Susan Montagu (daughter of the William Montagu, 5th Duke of Manchester), Hay joined the Royal Navy in 1840. He was posted to the sixth-rate HMS Vestal on the China Station in 1842 and saw action during the First Opium War. Promoted to lieutenant on 19 December 1846, he joined the steam frigate HMS Spiteful at Woolwich that month before transferring to the second-rate HMS Powerful in the Mediterranean Fleet in April 1848. He was promoted to commander on 28 August 1851 and given command of the sloop HMS Wasp in the Mediterranean Fleet in August 1852; he went ashore with the Naval Brigade and took part in the defence of Eupatoria in November 1854 and the Siege of Sevastopol in Spring 1855 during the Crimean War. He was wounded in the latter engagement and was appointed to the French Legion of Honour, 5th Class and the Turkish Order of the Medjidie, 4th class for his services in the Crimea.
Promoted to captain – in recognition of his services at Eupatoria – on 27 November 1854 and, having been appointed a Companion of the Order of the Bath on 5 July 1855, Hay was given command of the fifth-rate HMS Forth in December 1855. Entering politics, he became Liberal Member of Parliament for Wick in the 1857 general election and served his constituents there until the 1859 general election. Returning to sea, he became Captain of the paddle frigate HMS Odin on the East Indies and China Station in September 1859 and took part in the Battle of Taku Forts in August 1860 during the Second Opium War. From 1861 he served as commodore on the East Indies and China Station.
Hay became Member of Parliament for Ripon in April 1866 and served as Civil Lord of the Admiralty until the Liberal Government fell in June 1866; he went on to be Junior Naval Lord in December 1868. He resigned his seat in Parliament in February 1871 and was given command of the ironclad ram HMS Hotspur.
Promoted to rear admiral on 7 May 1872, Hay became Second-in-command of the Channel Squadron in January 1875, hoisting his flag in ironclad battleship HMS Northumberland and then the ironclad battleship HMS Black Prince, and then became Commander-in-Chief of the Channel Squadron, hoisting his flag in the armoured frigate HMS Minotaur, in November 1877. Promoted to vice-admiral on 31 December 1877, he was sent to the Mediterranean in July 1878 to take control of Cyprus and to occupy it in accordance with decisions reached at the Congress of Berlin.
Hay became Second Naval Lord in April 1880 and, having been advanced to Knight Commander of the Order of the Bath on 24 May 1881, he went on to be Commander-in-Chief, Mediterranean Fleet, hoisting his flag in the central battery ship HMS Alexandra in February 1883. He was promoted to full admiral on 8 July 1884 and, in his role as Commander-in-Chief, provided support for the Nile Expedition to relieve Major General Charles Gordon.
In a highly political appointment, Hay was made First Naval Lord in March 1886 when the Marquis of Ripon became First Lord of the Admiralty but had to stand down just five months later when William Gladstone's Liberal Government fell from power in August 1886. He was advanced to Knight Grand Cross of the Order of the Bath on 30 July 1886. He became Commander-in-Chief, Plymouth in May 1887, and having been promoted to Admiral of the Fleet on 15 December 1888, he retired in August 1892. He died at his home, Fulmer Place, at Fulmer in Buckinghamshire on 4 May 1916.
In 1876 Hay married Christina Lambert, youngest daughter of Nathaniel Grace Lambert, MP, of Buckinghamshire, who represented that constituency as a Liberal from 1868 to 1874; their daughter Minnie Christine Brenda Hay went on to marry Lord Aberdour.