Fox was born in London, the son of theatrical agent Robin Fox and actress Angela Worthington. He is the brother of actor Edward Fox and the film producer Robert Fox. His maternal grandfather was playwright Frederick Lonsdale. Like several members of the Fox family, he was educated at Harrow School. Like his brother Edward, after leaving Harrow, Fox took a short service commission in the Coldstream Guards.
Fox first appeared on film in The Miniver Story in 1950. His other early film appearances were made under his birth name, William Fox.
In 1962, Fox was working in a bank when Tony Richardson offered him a minor role in the film The Loneliness of the Long Distance Runner. Fox's father attempted to forbid this, claiming that his son had no talent for acting and that it would disrupt his life for him to give up his job in the bank, nevertheless Fox took the part.
During the 1960s, Fox gained popularity. In 1964, he won a BAFTA Award for Most Promising Newcomer for his role in The Servant (1963). His films included Those Magnificent Men in their Flying Machines (1965), King Rat (1965), The Chase (1966), Thoroughly Modern Millie (1967), Isadora (1968), and Performance (1970).
After finishing work on Performance (1970), Fox suspended his acting career.
In a 2008 interview, he said: "It was just part of my journey...I think my journey was to spend a while away from acting. And I never lost contact with it – watching movies, reading about it ... so I didn't feel I missed it."
He became an evangelical Christian, working with the Navigators and devoting himself to the ministry. During this time, the only film in which Fox appeared was No Longer Alone (1978), the story of Joan Winmill Brown, a suicidal woman who was led to faith in Jesus Christ by Ruth Bell Graham.
After an absence of almost ten years from mainstream cinema, Fox returned to the screen, appearing in Stephen Poliakoff's Runners (1983), A Passage to India (1984), Comrades (1986) and playing Anthony Blunt in the acclaimed BBC play by Alan Bennett, A Question of Attribution (1992). He also portrayed the character of Colonel Ferguson in Farewell to the King and the pro-German aristocrat Lord Darlington in The Remains of the Day (1993).
More recently he has appeared in the 2000 film Sexy Beast, the 2001 adaptation of The Lost World as Prof. Leo Summerlee, Agatha Christie's Poirot – Death on the Nile (2004) as Colonel Race and Charlie and the Chocolate Factory (2005) playing Mr. Salt, Veruca Salt's father. He appeared in the Doctor Who audio drama Shada, and in 2007, he guest-starred in the British television crime series Waking the Dead. He also appeared opposite his son Laurence Fox in "Allegory of Love", an episode in the third series of Lewis. He was part of the cast of Sherlock Holmes, as Sir Thomas, leading member of a freemason-like secret society.
In 2010, he filmed Cleanskin, a terrorist thriller directed by Hadi Hajaig, and in 2011 he played King George V in Madonna's film W.E.
He married Mary Elizabeth Piper in 1973, with whom he has five children: actors Laurence, Lydia, and Jack Fox, and sons Robin and Thomas. In 2007, Laurence and Lydia married actors Billie Piper and Richard Ayoade, respectively. Fox is grandfather to Laurence's two sons with Piper and Lydia's two daughters by Ayoade. His niece Emilia Fox is also an actress. Both Emilia and James Fox have portrayed roles in BBC's hit TV series Merlin and Fox has guest-starred in his son Laurence's television series Lewis.
In her 2014 book A Story Lately Told, Anjelica Huston refers to a sexual relationship with Fox when she was a teenager.