Gray was born in Windsor, Ontario, the son of Fannie (née Lifitz), a nurse, and Harry Gray, who had a business selling yard goods. His parents were both from Belarusian Jewish families. Gray attended Victoria School and Kennedy Collegiate Institute in Windsor before receiving a Bachelor of Commerce degree in 1952 from McGill University. He studied at Osgoode Hall Law School, where he received a Bachelor of Laws degree and was called to the bar, becoming a member of the Law Society of Upper Canada. On July 23, 1967, Gray married Sharon Sholzberg, also a lawyer. They had two children together: Jonathan David and Elizabeth Anne.
Gray was first elected to Parliament for the riding of Essex West on June 18, 1962, as a member of the Liberal Party of Canada. He was re-elected in twelve subsequent federal elections, making him the longest continuously-serving Member of Parliament in Canadian history.
Gray served in a variety of roles during his parliamentary career, including cabinet ministries and committee chairmanships during the Liberal governments of Lester Pearson and Pierre Trudeau, and as Opposition House Leader from 1984 to 1990.
From February 6, 1990, to December 21, 1990, he was Leader of the Opposition, between the resignation of John Turner as Liberal leader and the election to Parliament of his successor, Jean Chrétien.
When the Liberals returned to power after the 1993 election, Gray was appointed Leader of the Government in the House of Commons and Solicitor General of Canada. On June 11, 1997, he was appointed Deputy Prime Minister.
Gray also retained an interest in provincial politics in the Windsor area. In 1996, he was named as honorary co-chair of Dwight Duncan's bid to the lead the provincial Liberal Party. Duncan had previously worked in Gray's office.
Gray retired from Parliament on January 14, 2002, and was appointed Canadian Chair of the International Joint Commission, a bilateral organization which deals with Canada-United States trans-boundary issues on water and air rights.
On November 28, 2008, Carleton University announced that Gray had been appointed as the university's 10th chancellor. He died at a hospital in Ottawa on April 21, 2014, aged 82.
On January 15, 2002, then-Governor General of Canada Adrienne Clarkson granted Gray the title "The Right Honourable", in honour of his distinguished and record-setting contribution to Canadian political life. In 2003, he was made a Companion of the Order of Canada, a designation which can be bestowed on only 165 outstanding Canadians at any given time, in recognition of being "an enduring force in Canadian politics". He was a recipient of the Canadian Centennial Medal, the Queen Elizabeth II Silver Jubilee Medal, the 125th Anniversary of the Confederation of Canada Medal, Queen Elizabeth II Golden Jubilee Medal and the Queen Elizabeth II Diamond Jubilee Medal. He received honorary degrees from the University of Windsor, Assumption University (Windsor), Catholic University of Lublin (Poland), McGill University, and the University of Ottawa, and Honorary Lifetime Membership as Governor #71 with Junior Chamber International Canada (JCI Canada). In 2009, he became an honorary brother of Alpha Epsilon Pi.
The upgraded Windsor-Essex Parkway has been renamed the Right Honourable Herb Gray Parkway.
Gray was diagnosed with esophagus cancer in 1996 and recovered after radiation therapy. In 1999, he had an operation to treat a prostate condition unrelated to the cancer. In August 2001, Gray underwent valve replacement surgery to correct a heart condition he had known about for years.
Note: Canadian Alliance vote is compared to the Reform vote in 1997 election.