Born in Henniker, Merrimack County, New Hampshire, he was the son of William and Frances M. Shepard Patterson.
Patterson pursued classical studies, graduated from Dartmouth College in 1848, and was principal of the Woodstock Academy in Connecticut for two years. He attended the Theological Seminary at New Haven, Connecticut, where he studied law.
He married Sarah Parker Wilder and they had two children, George Willis Patterson and Arthur Hubert Patterson.
Patterson was a professor of mathematics, astronomy, and meteorology at Dartmouth College from 1854 to 1865.
Patterson was a member of the New Hampshire House of Representatives in 1862.
Elected as a Republican to the Thirty-eighth and Thirty-ninth Congresses Patterson was a United States Representative for the third district of New Hampshire from (March 4, 1863 - March 3, 1867). He was elected to the U.S. Senate and served from March 4, 1867, to March 3, 1873.
In the Senate he was chairman of the Committee on Enrolled Bills during the Forty-first Congress and a member of the Committee on the District of Columbia during the Forty-first and Forty-second Congresses.
In 1873, Patterson was found to have given false testimony to both House and Senate Committees who recommended his expulsion from the Senate for bribery in the Crédit Mobilier Scandal. Patterson's term expired before further action could be taken.(1873)
Patterson was a regent of the Smithsonian Institution and in 1877-1878 was again a member of the State house of representatives. He was State superintendent of public instruction from 1881 to 1893, and president of American Institute of Instruction.
Patterson died in Hanover, Grafton County, New Hampshire, on May 4, 1893 (age 69 years, 306 days). He is interred at Dartmouth College Cemetery, Hanover, New Hampshire.
The Patterson School, which was merged with the Garnett school in 1929 and then became Shaw Middle School at Garnett-Patterson, in Washington, DC was named in his honor because he sponsored the legislation creating a public school system for black students in Washington, DC. It was closed in 2013.