He followed his father in constructing marine steam engines and in 1854 established the compound engine as its most efficient form, which made the use of steamships more competitive than sail (also see Maritime history#Age of Steam). He took out numerous patents and expanded the business to employ thousands of workers.
John's father, David Elder, settled in Glasgow, and entered the establishment of Mr. Robert Napier, the well-known shipbuilder, under whom, in 1822, he constructed the first marine engine, which was fitted up in the River Leven for the passage between Glasgow and Dumbarton. David Elder, was the author of many inventions and improvements in the machinery of steam vessels, and to the excellence of his engines the success of the Cunard Line of steamers, in establishing regular communication between the opposite shores of the Atlantic, was mainly due. He died in January 1866, in his eighty-second year. John Elder was David's third son. John was also the brother of Alexander Elder, co-founder of Elder, Dempster & Co Ltd.
John Elder was educated at the High School of Glasgow, where he showed great excellence in mathematics and in drawing. After a five years' apprenticeship to Mr. Robert Napier, and a brief time passed in English engine works, he was placed at the head of the drawing office in Napier's works.
In 1852, he became a member of the firm of Randolph, Elliott, & Co., a firm that had been successful as millwrights, but had not attempted anything as marine engineers. In 1860 they began shipbuilding under the firm of Randolph, Elder, & Co.; in 1868, on the expiry of the copartnery. Elder continued the business, which reached a very great degree of prosperity. He soon became known as an engineer of singular ability. The company later became the Fairfield Shipbuilding and Engineering Company in 1886.
The greatest service which Elder rendered to practical engineering was the adoption of the compound or combined high and low pressure engines. Various attempts at this combination had been made before, but they had failed, owing to causes which engineers either did not understand or could not overcome. Where they had failed, Elder succeeded. Professor Macquorn Rankine, who has gone into all the details of the subject in his memoir of Elder, says that only one who had thoroughly studied and understood the principles of thermodynamics could have achieved this. A saving of fuel amounting to thirty or forty per cent was effected.
Elder took out many patents for improvements in marine machinery. Of some of his improvements he gave an account in papers presented to the British Association at Leeds in 1858, Aberdeen 1859, and Oxford 1860. In 1868, he read a paper before the United Service Institute in London on an improved form of warship, entitled 'Circular Ships of War, with immersed motive power.' In 1869 he was unanimously chosen president of the Institution of Engineers and Shipbuilders in Scotland, but died before he could take office.
Some idea of the magnitude of his business may be formed from the fact that when in business by himself he employed four thousand men, and that from June 1868 to the end of 1869 the number of sets of engines made by him was eighteen, their aggregate horse power 6,110, the number of vessels built fourteen, their aggregate tonnage 27,027.
During 1869, he was ill for several months. He proceeded to London to get the best advice, but while there he was cut off by disease (17 September 1869) of the liver at the early age of forty-five. Elder had married Isabella née Ure in 1857, daughter of A. Ure, esq., of Glasgow.
Isabella Elder, after her husband's death, besides adding largely to the endowment of the chair of civil engineering and applied mechanics at the University of Glasgow, also provided an endowment to create the John Elder Professorship of Naval Architecture and Ocean Engineering at the university, and also financed the establishment of the Queen Margaret College. In June 1901, she received an honorary Doctor of Laws (LL.D.) from the University of Glasgow.
Elder, as Professor Rankine remarks, was a genius in engineering. In person he was remarkably handsome, and in manner and character very attractive. He was quick and energetic in all his movements, full of resource, and remarkably enterprising. His character stood very high. Dr. Norman Macleod and others who knew him intimately pronounced him one whose great aim was to translate the facts of Christ's life into his own, especially in matters of common life. With his workpeople he was on the best of terms. He was much interested in schemes for their social, intellectual, and religious welfare; organised and contributed largely to a sick fund, and was contemplating the erection of schools and model houses on a large scale, when death ended his career. After his death the men in his employment, in begging to be allowed to attend his funeral, testified to his many virtues as a master. The intelligent and considerate spirit in which he looked on the struggles of the working class, while at the same time fully realising both the rights and responsibilities of employers, led to the belief that in his hands the problem of the relations of capital and labour would have found a solution acceptable to all. His death at so early an age was counted a great calamity, while the multitude that attended his funeral, and the silence of all the workshops in the neighbourhood as his body was carried to its resting-place, showed how much he was esteemed by all classes in his native city.
In 2012, he was inducted into the Scottish Engineering Hall of Fame.