Ruthven was born in Ireland, the elder son of Major the Honourable Patrick Hore-Ruthven, only surviving son of the 1st Baron Gowrie and his wife Lady Gowrie. His mother was Pamela Margaret Fletcher (who later married Major Derek Cooper. His younger brother is Malise Ruthven.
His father was killed in action in 1942, at which point he became his grandfather's heir apparent. When his grandfather, who had been the Governor General of Australia, was created Earl of Gowrie in 1945 he became known by the courtesy title Viscount Ruthven of Canberra. He was educated at Eton and Balliol College, Oxford, and later at Harvard University. He succeeded in the earldom of Gowrie on the death of his grandfather on 2 May 1955, and also succeeded his great-uncle (his grandfather's elder brother) the 10th Lord Ruthven of Freeland as 3rd Baron Ruthven of Gowrie on 16 April 1956 (the Scottish lordship of Ruthven of Freeland passed instead via the female line). Lord Gowrie inherited Castlemartin House and Estate at Kilcullen, County Kildare, Ireland, from his great-aunt Sheelagh Blacker in 1967, and later sold it to Tony O'Reilly.
Gowrie joined the Conservative front bench under Ted Heath in 1972 as a Lord-in-Waiting, a post he held until 1974. He later served under Margaret Thatcher as Minister of State for Employment between 1979 and 1981, as Minister of State for Northern Ireland between 1981 and 1983 at the Northern Ireland Office. In 1983 he was sworn of the Privy Council and entered the cabinet as Minister for the Arts, which he remained until 1985. He was also Chancellor of the Duchy of Lancaster between 1984 and 1985. Despite being offered the post of Secretary of State for Education and Science he resigned from the Cabinet in 1985, stating that it was impossible for him to live in London on the £33,000 salary provided for the post.
After leaving government, he became Chairman of Sotheby's (1985–1994) and later of the Arts Council of England - described as "the appointment of a Scot, born in Ireland and living in Wales". At the Arts Council, he secured the role as a distributor of funds from the National Lottery.
Lord Gowrie is a patron of the Elton John AIDS Foundation.
Lord Gowrie published one volume of poetry in his 20s, after a period working as an assistant to American poet Robert Lowell, and later co-authored a book on British painting, published in 1975. In the summer of 1999, having been diagnosed with a serious heart condition, he checked into Harefield Hospital, and, after a heart transplant, and a long recovery, left hospital in 2000. His health has remained frail since. He became friends with his principal surgeon, Sir Magdi Yacoub, and now chairs the institute named for him. Following his release from hospital, he published his first book of poetry for decades, "The Domino Hymn," which contains references to his illness. He was elected in 2003 a Fellow of the Royal Society of Literature In January 2009, Grey Gowrie accepted Farad Azima's invitation to chair the Advisory Board of the Iran Heritage Foundation.
Lord Gowrie married Alexandra Bingley, daughter of Colonel Robert Bingley, on 1 November 1962. They had one son:(Patrick Leo) Brer Ruthven, Viscount Ruthven of Canberra (b. 4 February 1964)
Lord Gowrie and Alexandra Bingley divorced in 1974. On 2 November 1974, he married Adelheid Gräfin von der Schulenburg (b. 24 October 1943), sixth and youngest child and fifth and youngest daughter of Fritz-Dietlof von der Schulenburg (1902–10 August 1944) and his wife Charlotte Kotelmann.