He was the son of the engineer Walter Neilson, a millwright and later engine wright, who had been a partner of David Mushet in Calder Ironworks, Glasgow. He was born in Shettleston and was trained as an engine wright. After the failure of a colliery at Irvine he was appointed foreman of the Glasgow Gasworks in 1817 at the age of only 25. Five years later he became the manager and engineer there, a position he would hold for 40 years.
While trying to solve a problem with a blast furnace at Wilsontown Ironworks, Neilson realized that the fuel efficiency of the furnace could be increased by blowing it with hot air, rather than cold air, by passing it through a red-hot vessel. Experiments were continued at Clyde Iron Works, leading to his forming a partnership with Charles Macintosh and others to exploit it. Patents were obtained for the system in 1828.
Experimentation showed that a temperature of 600° Fahrenheit reduced consumption to a third of that with cold blast, and enabled raw coal to be used instead of coke, with a further cost saving. It also enabled the exploitation of black band ironstone, the use of which had previously proved unprofitable.
In the early 1830s litigation was successfully conducted against those who adopted his methods without licence. After that, Neilson and his partners licensed it widely at one shilling per ton iron made, a level low enough to discourage evasion. The royalties were initially low, but by 1840 were producing £30,000 per year from 58 ironmaste.
Certain infringers were intransigent. Between 1839 and the expiry of the patent in 1842 a considerable number of proceedings were brought. Neilson v Baird was heard in the Court of Session in 1843, in a trial lasting 10 days and costing £40,000. Further proceedings against Baird ended in the award of damages of £160,000.
Neilson retired from Glasgow Gasworks in 1847. He bought an estate on Bute. Later he bought an estate at Queenshill, near Kirkcudbright. There he died. His son, Walter Montgomerie Neilson, erected a monument to his memory there in 1883.
Both in Glasgow and near Kirkcubright, he founded institutions for the education of working men.
William Neilson, James's brother, founded the Glasgow engineers and locomotive manufacturers Neilson and Company, in 1836, partly financed by James. James's son Walter took over the running of the firm in 1843.