James Potter was of Scots descent, born in County Tyrone, Ireland. He came to Colonial America with his father, John Potter, in 1741, and the family settled in Cumberland County, Pennsylvania, where his father became high sheriff in 1750. His first wife was Elizabeth Cathcart of Philadelphia, and his second wife was Mary Patterson Chambers, daughter of James Patterson of Mifflin County. His daughter Martha was married to Andrew Gregg.
As a militia lieutenant for Northumberland County, where he made his home, James Potter took part in the Kittanning Expedition during the French and Indian War and reached the rank of lieutenant-colonel by the end of the war. He also served during Pontiac's Rebellion. In 1776, he served as a colonel and as a member of the first Pennsylvania State Constitutional Convention.
During the American Revolutionary War, he led militia troops at the battles of Trenton, Princeton, Brandywine and Germantown. In April 1777, he was promoted to brigadier-general of the Pennsylvania militia. In 1777, with the troops under his command in the counties of Philadelphia, Chester, and Delaware, he obtained important information for George Washington, and prevented supplies from reaching the enemy. On 11 December, while the army under Washington was on its way to Valley Forge, after part of it had crossed the Schuylkill River at Matson's Ford, it was found that the enemy under Cornwallis were in force on the other side. "They were met," writes Washington, "by General Potter, with part of the Pennsylvania Militia, who behaved with great bravery, and gave them every possible opposition until he was obliged to retreat from their superior numbers." On this same day a group of Pennsylvania Militia under the command of General Potter defended the home of Thomas Wynne of Blockley from a band of British marauders. The graves of three British soldiers killed are on the Wynnestay property. After this day of service, Potter took a leave of absence to care for his sick wife. Accompanying General Potter through the Revolutionary War was his servant Hero Wade.
Potter's service, which had been effective at interdicting trade and trafficking between the countryside and British-occupied Philadelphia, was missed. In the spring of 1778, Washington wrote from Valley Forge: "If the state of General Potter's affairs will admit of his returning to the army, I shall be exceedingly glad to see him, as his activity and vigilance have been much wanted during the winter." Potter eventually returned to service in late May 1778, but the role of eastern Pennsylvania in the conflict declined after the British left Philadelphia for New York in June.
When not engaged in military action, Potter was a farmer. Before the Revolution he was involved in opening former Indian territories to European settlement, and warranted a large tract of such land for his own use.
In 1780, Potter was elected to Pennsylvania's Council of Censors or Supreme Executive Council. In that year he lost a vice-presidential election to William Moore, earning only six of the sixty votes cast. (The position of vice-president is analogous to the modern office of lieutenant governor.) However, the following year, on 14 November 1781, he was elected to the vice-presidency, earning thirty-eight votes to James Ewing's twenty-eight. But on 7 November 1782, he lost both the presidential race to John Dickinson (earning only thirty-two votes to Dickinson's forty-one) and the vice-presidential race to James Ewing (earning thirty-four votes to Ewing's thirty-nine). During his vice-presidential term he served as an ex officio member of the board of trustees of the University of Pennsylvania.
Potter was named a major-general of the militia in 1782, and, in 1785, became deputy surveyor in Northumberland County.
James Potter died near Greencastle, Franklin County, Pennsylvania, in 1789 as the result of a construction injury during a barn raising in Penns Valley. He left Penns Valley and travelled to Greencastle, seeking medical help from his family doctor. His oldest daughter also lived there. He was a resident of Mifflin County at his death, and was one of Pennsylvania's largest landowners. The area of his plantation is now in Centre County. The Potter-Allison Farm was listed on the National Register of Historic Places in 1977.
Several place names in Pennsylvania, such as Potter County and Potters Mills (in Centre County), were given in his honor.
A Pennsylvania Historical Marker near Potters Mills commemorates a Revolutionary-era fort built by General Potter. The marker was erected along State Route 144 in 1947. Its text reads:
Potter's Fort Built 1777 by Gen. James Potter. A stockaded fort refuge for the settlers of the valley region. The site is on the nearby rise.