Gravey started performing at the early age of five, under his father's direction.
Prior to World War I, he received an education in Britain and could speak both French and English fluently, something which would become useful in his movie roles. During the war, Gravey served in the British Merchant Marine Corp.
In 1936, he married the French actress Jane Renouardt, who was fifteen years his senior. They remained together until his death, on 2 November 1970, of a heart-attack. Jane died on 3 February 1972. They had no children.
Gravey performed in four films in 1913 and 1914 (as Fernand Mertens), but his first film of importance was L'Amour Chante, released in 1930. In 1933, he made his first English language movie, Bitter Sweet, which would become more famous in its 1940 incarnation with Jeanette MacDonald and Nelson Eddy.
In 1937, after several more French and British movies, Gravey went to Hollywood, where the spelling of his last name was altered to 'Gravet', and he became the focus of a rather extensive Hollywood publicity campaign (instructing moviegoers to pronounce his name properly: "Rhymes with 'Gravy'"). Unfortunately for Gravey, he was offered only standard parts, the type of Gallic-lover roles that Louis Jourdan would play in the 50's and 60's.
The first two films he made in Hollywood were for Warner Brothers: The King and the Chorus Girl (1937), opposite Joan Blondell and Jane Wyman, and Fools for Scandal (1938), opposite Carole Lombard and Ralph Bellamy. Gravey then signed with Metro-Goldwyn-Mayer and was cast as Johann Strauss in the expensive biopic The Great Waltz, opposite Luise Rainer and Miliza Korjus.
MGM next planned to star Gravey in a film version of Rafael Sabatini's adventure novel Scaramouche, but instead he returned to France just before the Nazi occupation began. Although he had agreed to appear in German-approved French films, Gravey was also an underminer of the invaders as a member of the French Secret Army and the Foreign Legion.
At the end of the war, Gravey was considered a war hero, and continued to star in such French productions as La Ronde (with Danielle Darrieux), and Royal Affairs in Versailles (1954). Among his last English language performances were How to Steal a Million (1966), Guns for San Sebastian (1968) and The Madwoman of Chaillot (1969), in which he played the police inspector.