Born at Cromarty, Scotland, in 1848. He was the eldest son of Captain John R. Ross (d.1889), merchant and shipowner; and his wife Mary B. McKeddie (1826–1896), daughter of Captain McKeddie, of Newcastle-on-Tyne. Ross was educated at Inverness Royal Academy and afterwards trained as a civil engineer in England. He worked for a while in a railway, harbour and water works before coming to the United States in 1868 to apply his talents to the rapidly expanding North American railway industry. In 1870, he was appointed engineer, then chief engineer of the Ulster and Delaware Railway.
After marrying in 1872 (see notes below on his family), Ross became chief engineer of the Wisconsin Central Railway and then the Lake Ontario Shore Railroad. On Lake Ontario, Ross came into contact with George Laidlaw and several other well-connected railway promoters who persuaded him to come to Canada. He was appointed chief engineer to Laidlaw's Victoria Railway and in 1879 he built the Credit Valley Railway. He was then appointed acting consulting engineer for the Ontario and Quebec Railway, during which time he came into contact with three ambitious young men, namely William Mackenzie, Donald Mann and Herbert S. Holt.
After slow progress was being made in connecting the Canadian Pacific Railway west of Winnipeg, in 1883 under new CEO William Cornelius Van Horne, the company formed the wholly owned North American Railway Contracting Company (NARCC). With the brief to meet up with the team driving east from the Pacific Ocean under Andrew Onderdonk, Ross was appointed General Manager and Chief Engineer of the NARCC, and immediately employed the services of Mackenzie, Mann and Holt. Starting at Swift Current, they built 623 miles (1,003 km) of railway to Craigellachie, British Columbia, by November 7, 1885, over the Rocky Mountains, the Selkirks and the Gold Range. Completing the project a year ahead of time, Van Horne commented at the opening of the line that Ross's record meant millions to the Canadian Pacific Railway. In achieving this, Ross stuck up a lifelong friendship with Donald Smith, 1st Baron Strathcona and Mount Royal, who drove "The Last Spike".
In 1886, Ross was appointed manager of construction for the Ontario and Quebec Railway, filling in gaps to allow full access into the CPR's Montreal to Windsor line, and onwards to the Michigan Central Railroad. Ross then negotiated the entry of the CPR into the American state of Maine, building the International Railway of Maine east from Montreal into an Atlantic Ocean terminal in Bangor, with an extension to Saint John. Ross then completed extensions of the CPR west of the Rockies to enable full access to the Pacific. At the time of his death, he was still a director of the CPR and was said to have been the company's largest shareholder.
Having completed his works at the CPR, he advised both Lord Strathcona and William Mackenzie on railway contracts in South America, netting US$20Million alone for consulting works in Argentina and Chile. He also form a railway consulting and contracting company with Mackenzie, Mann and Holt, with:Ross as General Manager
Mann preparing and grading roadbeds
Mackenzie cut the ties and organised timber work for trestles and bridges
Holt laid track, general finishing, and clean-up work
Advising and constructing on feeder lines north from the CPR mainline, the company completed early works on the Winnipeg and Hudson Bay Railway, and constructed both the Regina and Long Lake Railway and the Calgary and Edmonton Railway. The partners also negotiated the incorporation of several land development companies, including the Calgary and Edmonton Land Company and the Canada Land and Investment Company. In a later 1887 partnership with George Stephen, 1st Baron Mount Stephen and Van Horne, they established the Lake of the Woods Milling Company, which bought and processed grain, and in 1889 Ross became first president of the Columbia River Lumber Company, which provided timber for railways and housing projects.
After moving to Montreal in 1888, in partnership with William Mackenzie, he oversaw the electrification of street railways in Montreal, St. John, the Toronto Street Railway and Winnipeg Transit. They later took over the City of Birmingham Tramways Company Ltd in England, reorganising and electrifying that.
Mackenzie then looked for other opportunities in South America, resulting in similar projects in Mexico City and the highly profitable São Paulo Tramway, Light and Power Company, whose holding company later acted as a holding company for all of the teams Canadian and global street car investments.
As the partners recognised the need for clean electricity, they each became involved in local hydro electric projects. Ross through investment became first president of the Mexican Power Company, which developed a hydro electric dam at Necaxa to provide electricity for Mexico City.
In 1890 Ross replaced Job Abbott as president of Dominion Bridge Company, a major contractor to the CPR for replacing wooden bridges with stronger and lower maintenance steel replacements. Although credited to the drive of Ross, vice-president James Pawley Dawes lead the developed via joint-venture the St Lawrence Bridge Company to construct the Quebec Bridge.
Ross saw the great need for steel, and formed a syndicate to buy the Dominion Coal Company. Investing in further shares independently, Ross built up such a huge stake in Dominion Coal that he was invited to join the board of the Dominion Iron and Steel Company. However, his influence could not resolve a long contract dispute between the two for the supply of high-grade coal at a disadvantageous price, which resulted in a case review at the High Court of Justice in London, England. Although fault was found on both sides, the contract was found to be legal. Ross resigned from both boards, thus allowing a later merger between the two companies.
In 1892, architect Bruce Price completed a French Château-style mansion in Montreal's Golden Square Mile, on Peel Street for the Rosses. Between 1897 and 1912, the Maxwell Brothers expanded the house, adding an art gallery among other rooms. In 1910, their son, Jack Ross, had built his own home also on Peel Street, but after the death of his parents he moved back to his childhood home. He hired the firm Trowbridge & Livingston, again expanding and remodelling the house. When their son was declared bankrupt in 1928, the house was purchased by J.W. McConnell who donated it to McGill University, when it was renamed Chancellor Day Hall.
Ross had a passion for art and became a significant collector. At the time of his death, he had amassed one of the largest and finest collections of Old Masters on the North American continent. Through this passion, Ross became president of the Montreal Art Association, leaving much of his collection to the Museum on his death.
Ross' father had been a shipowner, and Ross too had become a keen sailor after settling in Montreal in 1888. He was made honorary Commodore of the Royal St. Lawrence Yacht Club. He was a member of the New York Yacht Club and the first Canadian member of the Royal Yacht Squadron and the Royal Thames Yacht Club, both in England. He owned several yachts during his lifetime, including the Glencairn, which won the coveted and prestigious Seawanhaka Corinthian Cup for half-raters in American waters, 1896. However his most famous yacht was the Liberty, purchased in 1912 from the deceased newspaper magnate Joseph Pulitzer. The yacht (which he renamed Glencairn after his previous vessel) required a staff of sixty-five and had a full auditorium, numerous state rooms, and first-class dining-rooms. He undertook a cruise around the world in the hope that it would restore his health, but died soon after returning to Montreal.
Retiring from engineering and active investment, Ross sat on the boards of numerous companies including the Bank of Montreal and the Royal Trust Company. He was a member of both the American and the Canadian Society of Civil Engineers and appointed honorary Lieutenant-Colonel of the 17th Duke of York's Royal Canadian Hussars. He was a Trustee of Bishop's College School, Governor of the Royal Victoria Hospital, Montreal, and of McGill University, donating generously to all institutions. Ross was a member of several clubs not already mentioned, including the Constitutional Club in London; Manhattan Club in New York City and the Montreal Hunt, Jockey, Royal Golf and Racquet Clubs.
Throughout his lifetime, and in his will, Ross made numerous donations to various charitable and cultural institutions. In memory of his parents, at Lindsay, Ontario, he built the Ross Memorial Hospital and a Nurse's Home. During his lifetime he had donated generously to the Alexandra Hospital and the Royal Victoria Hospital, both in Montreal. In his will, he directed his son, J.K.L. Ross, to continue his support of the Royal Vic, by way of which his son gave $1 million to construct the Ross Memorial Pavilion in 1915. He also built the Protestant Hospital for the Insane at Verdun, Quebec, of which he became a Governor.
Ross was a generous donator towards McGill University and he made several donations of $25,000 during his lifetime to the Montreal Art Association and towards the building of the Montreal Museum of Fine Arts in 1912. In his will, he left the Art Association a further $100,000 and much of his collection of Old Masters.
In 1872, James Ross had married Annie Kerr (1847–1915), the eldest daughter of John W. Kerr (1824–1904), of Kingston, New York, and his wife Annie Jane Love. His father-in-law was a prominent politician with the Democratic Party and formerly the Sheriff of Ulster County, New York. The Rosses were the parents of one son,John Kenneth Leveson Ross. In 1902, he married Ethel Matthews, daughter of Wilmot DeLoui Matthews (1850–1919), of Toronto. They were the parents of a son and a daughter. They divorced in 1930 and the following year, at Jamaica, he married Iris de Lisser, sister of H. G. de Lisser.
Towards the end of his life, Ross devoted most of his time to sailing in European and Canadian waters. Following his around the world trip aboard his yacht Glencairn, Ross died of existing heart complications at his home in the Golden Square Mile on September 20, 1913.