Lord Elgin was the son of the 7th Earl of Elgin and 11th Earl of Kincardine and his second wife.
He shared his birthday 20 July with his father. He had seven brothers and sisters and four half-sisters and one half-brother from his father's first marriage. Lord Elgin's father was reportedly impoverished by the purchase of the Elgin Marbles. His father had acquired them at great expense, but sold them to the British government for much less.
James Bruce was educated at Eton College and Christ Church, Oxford, graduated with a first in Classics in 1832. While at Oxford, he became friends with William Ewart Gladstone.
He was elected at the 1841 general election as a Member of Parliament for Southampton, but the election was declared void on petition. He did not stand in the resulting by-election.
James Bruce became Governor of Jamaica in 1842, and in 1847 was appointed Governor General of Canada.
Under Lord Elgin, the first real attempts began at establishing responsible government in Canada. Lord Elgin became the first Governor General to distance himself from the affairs of the legislature. Since then, the Governor-General has had a largely symbolic role with regards to the political affairs of the country. As Governor-General, he wrestled with the costs of receiving high levels of immigration in the Canadas, a major issue in the constant debate about immigration during the 19th century.
In 1849 the Baldwin-Lafontaine government passed the Rebellion Losses Bill, compensating French Canadians for losses suffered during the Rebellions of 1837. Lord Elgin granted royal assent to the bill despite heated Tory opposition and his own misgivings over how his action would be received in England. The decision sparked the Burning of the Parliament Buildings in Montreal by an English-speaking mob. Elgin was assaulted. Instead of calling in the military, he withdrew his family to their country residence and allowed civil authorities to restore order. The French-speaking minority in the Canadian legislature also unsuccessfully tried to have him removed from his post.
In 1849, the Stony Monday Riot took place in Bytown on Monday 17 September. Tories and Reformists clashed over the planned visit of Lord Elgin, one man was killed and many sustained injuries. Two days later, the two political factions, armed with cannon, muskets and pistols faced off on the Sappers Bridge. Although the conflict was defused in time by the military, a general support for the Crown's representative, triumphed in Bytown (renamed Ottawa by Queen Victoria in 1854). In 1854, Lord Elgin negotiated the Reciprocity Treaty with the United States in an attempt to stimulate the Canadian economy. Later that year, he granted royal assent to the law that abolished the seigneurial system in Quebec, and then resigned as Governor-General.
In 1857, Lord Elgin was appointed High Commissioner and Plenipotentiary in China and the Far East to assist in the process of opening up China and Japan to Western trade. During the Second Opium War, he led the bombardment of Canton (Guangzhou) and oversaw the end of the war by signing the Treaty of Tientsin (Tianjin) on 26 June 1858.
In June 1860, Lord Elgin returned to China to assist with additional attacks, which were initially led by his brother. On 18 October 1860, not having received the Chinese surrender and wishing to spare the imperial capital of Peking (Beijing), he ordered the complete destruction of the Old Summer Palace (Yuanming Yuan) outside the city in retaliation for the torture and execution of almost 20 European and Indian prisoners, including two British envoys and The Times journalist Thomas Bowlby. The Old Summer Palace was a complex of palaces and gardens eight kilometres northwest of the walls of Beijing; it had been built during the 18th and early 19th centuries, and was where the emperors of the Qing dynasty resided and handled government affairs. An alternative account says that Lord Elgin had initially considered the destruction of the Forbidden City. However, fearing that this act might interfere with the signing of the Convention of Peking, which was where it was being negotiated, he opted for the destruction of the Old Summer Palace in its stead.
The Old Summer Palace was set aflame by 3,500 British troops and burnt for three days. Lord Elgin and his troops looted many treasures from the palace and took them to Britain. Attacks on the nearby Summer Palace (Qingyi Yuan) were also made, but the extent of destruction was not as great as to the Old Summer Palace. On 24 October 1860, Lord Elgin signed the Convention of Peking, which stipulated that China was to cede part of Kowloon Peninsula and Hong Kong in perpetuity to Britain.
In between Lord Elgin's two trips to China, he had visited Japan. In August 1858, he signed a Treaty of Amity and Commerce whose negotiation was much eased by the recent Harris Treaty between Japan and the United States. Lord Elgin was ambivalent about the British policy on forcing opium on the people in the Far East. It was not without internal struggle that he carried out the duty laid on him by Britain. In a letter to his wife, in regard to the bombing of Canton, he wrote, "I never felt so ashamed of myself in my life."
He became Viceroy of India in 1862, and was the first to use Peterhoff, Shimla as the official residence of the Viceroy. He died in 1863 of a heart attack while crossing a swinging rope and wood bridge over the river Chadly, on the lap between Kullu and Lahul. He was buried in the churchyard of St. John in the Wilderness in Dharamshala.
Elgin's first marriage to Elizabeth-Mary Cumming-Bruce on 22 April 1841 was short lived, his wife dying shortly after the birth of a second daughter on the 7 June 1843 in Jamaica.
Elgin's second wife, Lady Mary-Louisa Lambton, mother of the 9th Earl of Elgin was a daughter of the 1st Earl of Durham, a prominent author of the Report on the Affairs of British North America (1839) (as well as Lieutenant Governor of Lower Canada and Governor General of the Province of Canada), and niece of the Colonial Secretary the 3rd Earl Grey (who was uncle to Albert Grey, 4th Earl Grey, later Governor General of Canada)
In Canada, the towns of Kincardine, Ontario, Port Elgin, Ontario, and Bruce Mines, Ontario, are named for James Bruce. Also are Bruce County, Ontario, Elgin County, Ontario, the long Bruce Peninsula into Lake Huron, and the communities of Elgin, New Brunswick and Elgin, Nova Scotia. There are numerous Elgin Roads and Elgin Streets in Canada and in India. The Elgin Bridge in Singapore, and Elgin Street, Carlton, State of Victoria, and Elgin Street, Hong Kong are also named for Bruce, as is the Lord Elgin Hotel in Ottawa.
James Bruce's legacy in Canada was the subject of a National Film Board of Canada short docudrama, Lord Elgin: Voice of the People (1959), directed by Julian Biggs.