De Savary is the son of a French-born Essex farmer, and was educated in England at the private Charterhouse School from which he was expelled at the age of 16. He then moved to Canada where his divorced mother and stepfather lived and did gardening, baby-sitting and children's private tuition. At the age of 18, with his wife Marcia and daughter, he moved back to the UK to work for his father. During a visit to Canada in 1969 he took over a small import-export agency, Afrex, that did business in Africa. On a subsequent flight to Nigeria he met the brother of the President of Nigeria with whom he went into business supplying wheat, flour, steel, cement and other goods to Nigeria and other African countries making him a millionaire by the age of 30.
The bulk of his business career has been spent in the shipping and oil sectors; he once owned or managed 13 shipyards around the globe, still retaining one shipyard in the United Kingdom, and he still has a global oil-trading and refuelling business.
His first venture into hospitality was the St. James' Clubs in the late 1970s, in Los Angeles, London, Paris and Antigua, which he sold in the late 1980s to finance the £4m purchase of Skibo Castle. De Savary built up a large business empire in the 1980s, with property interests including Land's End and John o' Groats.
However, in the early 1990s economic downturn his empire collapsed – he sold both Land's End and John o' Groats in 1991 for an undisclosed sum to the businessman Graham Ferguson Lacey and his holding company Placeton went bankrupt in 1994 with debts of £200 million by one source and £715 million by another.
His business activities since 2000 concentrated on property development and hotels, with a number of major country house hotels incorporating golf courses. De Savary saw a niche for the affluent; leisure properties that were small enough to make guests feel as though they were on their own private estate, but equipped with all the facilities of the world's great hotels. His first such development was The Carnegie Club at Skibo Castle in Scotland, the venue for Madonna and Guy Ritchie's wedding. This was sold in 2003 to Ellis Short. Through his wife Lana's company, Havana West, other similar developments have included: the Cherokee Plantation in South Carolina; Stapleford Park and Bovey Castle, both in England; and Carnegie Abbey in Rhode Island. Each is a private club with golf courses and other amenities — clay pigeon shooting, falconry, horseback riding, tennis — depending on what fits with the club's local environment. Again with Lana's Havana West company he founded the Abaco Club at Winding Bay in Abaco, Bahamas, building a spectacular golf course on the beautiful Winding Bay beach and bluff.
He bought four properties in Grenada in the Caribbean, where he was developing a marina and resort village.
In late 2009, de Savary purchased Vanderbilt Hall, a mansion hotel located in Newport, Rhode Island. He added a small collection of American Illustration artworks to the property from the American Illustrators Gallery, New York, including a piece by Howard Chandler Christy titled "Jeanie with the Light Brown Hair". The painting depicts Stephen Foster composing the song of the same name. Other artists on display were Bradshaw Crandell, William Soare, Phil Berry, R. C. Kauffmann, Julian De Miskey, Constantin Alajalov, Helen Dryden, John Lagatta, George Hughes, Thomas Webb, Rico Tomaso, Gilbert Bundy, Hans Flato, Carl Burger, Rolf Armstrong and Earl Bergey. The property was sold to Grace Hotels in 2010.
He led the British sailing team in its challenge for the America's Cup in 1983 but his contender, Victory 83, was beaten by Australia II in the final heat.
He used the motoryacht Kalizma (formerly home to Elizabeth Taylor and Richard Burton during filming in London, named for their children) as a support vessel for the America's Cup races, but has since sold the ship. He also once owned the luxury yacht MY Land's End. He founded Pendennis Shipyard in Falmouth, Cornwall, which builds and restores luxury yachts. He is also a sponsor of the Grenada Sailing Festival. He has raced for many years in the Bucket Regatta in Newport, Rhode Island, and St Barts in the Caribbean. He was awarded the trophy "Spirit of the Bucket" in 2010.
In November 2005, he succeeded Theo Paphitis as chairman of Millwall Holdings plc and as chairman of Millwall F.C.. Stewart Till succeeded him on 3 May 2006 as the football club chairman, and de Savary remained as chairman of Millwall Holdings plc until October 2006.
In March 2011, de Savary was linked with a deal to purchase the financially stricken League One club Plymouth Argyle F.C.. However, de Savary denied any interest in buying the club, which was eventually purchased by Plymouth City Council the following October.
De Savary married three times (once to Alice Simms for six weeks) and has five daughters. Two are from his first marriage – Lisa Tomlinson, who worked in public relations in 1997 and as a photographer in 2010 and who has provided him with two grandsons and a granddaughter; Nicola, who studied medicine at King's College London, is a doctor and mother to three more grandsons. His third wife is Lana Paton, from Charleston, South Carolina, and the couple have three daughters – Tara; Amber, who as a dressage rider represented her country more than 20 times at dressage; and Savannah who studied at Oxford University. Tara and Amber have worked with their father in the family business,
In December 1987, after departing from St. Barthélemy in the Caribbean with his pilot, a nanny, his pregnant wife and his three daughters, their plane went into a stall, plunged into the Caribbean and landed upside down. The pilot died, and one of de Savary's daughters had to be revived on the beach. De Savary said, "At that point, my philosophy on life changed a little. When you genuinely look death in the eye, you know that nothing's going with you, and life is but a thread. It's a pretty tenuous thing we're hanging on to. So, what is the point of making money? I concluded it certainly isn't for accumulating it. That's the most stupid thing I ever heard of. So, there can be only one point, and that's to spend it. Now, I'm not ridiculously wasteful, but I may be slightly extravagant. As Andrew Carnegie said, to die rich is to die disgraced."