Witherell was born in Mansfield, Massachusetts. After completing preparatory studies, he served in the Continental Army from 1775 to 1783 during the American Revolutionary War. He entered service as a private and rose to the rank of Adjutant in the Eleventh Massachusetts Regiment. He was severely wounded in the Battle of White Plains in 1776.
After the war, Witherell studied medicine and law, and was licensed to practice medicine in 1788. He moved to Hampton, Vermont in 1788 and to Fair Haven in 1789 and continued the practice of his profession.
Witherell was a member of the Vermont House of Representatives from 1798 to 1802. He was associate county judge from 1801 to 1803, judge of Rutland County from 1803 to 1806, and an executive councilor from 1802 to 1806.
He was elected as a Democratic-Republican to the Tenth Congress, serving from March 4, 1807 until May 1, 1808, when he resigned to accept an appointment by President Thomas Jefferson as one of the Judges of the Supreme Court for the Territory of Michigan. While serving in Congress, he argued in favor of the Act that abolished the slave trade, and voted for the Act, which passed in 1808.
During the War of 1812, he was in command of the troops at Detroit in the absence of General William Hull, and was taken prisoner when General Hull surrendered. He lived in Fair Haven, Vermont while on parole from the British and later was exchanged and returned to his duties in Detroit. On April 30, 1821, Governor Lewis Cass and Judges John Griffin and James Witherell passed a new act that changed the name of the Catholepistemiad or University of Michigania to the University of Michigan, and put control in the hands of a Board of Trustees consisting of twenty members plus the governor.
After serving as a Supreme Court justice for nearly twenty years, Witherell resigned in 1828 to accept an appointment by President John Quincy Adams to become Secretary of the Territory. He held the position until May 1830. Witherell was Acting Governor of the Territory of Michigan for the first three months of 1830.
Witherell was married on November 11, 1790 to Amy Hawkins and the couple had six children. His son, Benjamin F. H. Witherell, was also a Justice on the Michigan Supreme Court, and his grandson was United States Senator from Michigan, Thomas W. Palmer.
Witherell died at his home in Detroit on January 9, 1838, and was interred at the Russell Street Cemetery. Later, he was reburied in Elmwood Cemetery in Detroit.