Miller was born in Quincy, Norfolk County, Massachusetts, in 1933, the son of Ellis Warren Appleton Miller (1898 – c. 1986), a bookkeeper, salesperson, and descendant of several Mayflower passengers; and Sophia "Sophie" June Squarebriggs (1899–1998), a Canadian-born former governess. Miller's ancestry is rooted in the British North American colonial elite, and descends over time, he claims, from Henry I of England, Louis IV the Carolingian, and William, King of the Scots.
Miller attended Cornell University School of Hotel Administration and graduated in 1955, with a Bachelor of Science in Hotel Administration. He was a member of the Phi Kappa Psi Fraternity and the Irving Literary Society.
The concept of "duty-free shopping" – offering high-end concessions to travelers, free of import taxes – was in its infancy when, with Chuck Feeney, Miller founded DFS on November 7, 1960.
DFS began operations in Hong Kong, where it still maintains its corporate headquarters, later expanding to Europe and other continents. DFS' first major breakthrough came in the early 1960s, when it secured the exclusive concession for duty-free sales in Hawaii and Hong Kong, allowing it to market its products to Japanese travelers. DFS eventually expanded to off-airport duty-free stores and large downtown Galleria stores, and became the world's largest travel retailer. In 1996, Feeney's interests were acquired by Louis Vuitton Moët Hennessy (LVMH), the French luxury goods group, for $1.63 billion.
Miller kept his shares and remains a significant minority shareholder (38%) in DFS alongside LVMH, which bought out Feeney's share. More recently, Miller targeted the emergence of mainland China's economic elite. To present its new merchandise, DFS has moved beyond the airport liquor store venue and into store environments, seeking to market the broadest portfolio of brand names in the travelling marketplace. In 2010, the DFS Group had an estimated sales of about €2.2 billion. As of 2012, it has 150 stores in 15 Pacific states. Its downtown activities operate under the Galleriera name.
Miller was named the November 2010 Personality of the Month by The Moodie Report in recognition of his advancement of the luxury goods industry's cause. That year, Duty Free Shops (DFS) celebrated its 50th anniversary at the DFS Galleria Sun Plaza store in Hong Kong, with a gala attended by 2000 guests, and featuring a performance by Wyclef Jean.
In 1973, Miller founded Search Investment Group Ltd, an international investment company of which he is CEO and chairperson. The Search investment Group Ltd is a large shareholder in Ortelius Capital Partners, which is an investment firm founded by Miller's son-in-law HRH Prince Pavlos of Greece and Mr. Peter DeSorcy. He is also principal of SAIL advisors (Search Alternative Investments Ltd), global investors in hedge funds. Along with other assets, Miller is currently ranked the 68th wealthiest person in the UK, down from 22nd in 2003.
In October 2003, Miller's monohull yacht, the Mari-Cha IV, set a new world record, becoming the first monohull ever to cross the Atlantic Ocean in under seven days.  During the same crossing, the yacht also broke the 24-hour distance record, by sailing 525.5 nautical miles (973.2 km). Both he and his son-in-law Pavlos, Crown Prince of Greece, son of King Constantine II of Greece were crew members.
On April 30, 2005, the Mari-Cha IV won the Rolex Transatlantic Challenge and broke the century-old record for a west-to-east transatlantic crossing established by Charlie Barr. It completed the 2,925-nautical mile passage across the North Atlantic between New York and England in a time of 9 days, 15 hours, 55 minutes and 23 seconds—a full 2 days, 12 hours, 6 minutes and 56 seconds faster than the record set 100 years earlier. During the record-breaking crossing, Miller pushed the Mari-Cha III beyond its limits in poor weather, and the mainsail headboard and the headboard cars on both mainsail and mizzen broke.
Miller is married to Ecuadorian-born María Clara "Chantal" Pesantes Becerra, they have three daughters: Pia (b. 1966), Marie-Chantal (b. 1968) (who married Pavlos, Crown Prince of Greece), and Alexandra (b. 1972)
He also owns the 36,000-acre (146 km2) Gunnerside Estate in Yorkshire, one of the largest sporting country estates in Britain.
Miller is a trustee of the Asia Society, the Independent non-executive director and Member of the Remuneration Committee of Hong Kong and Shanghai Hotels Ltd., and presidential councilor and trustee emeritus of Cornell University.