He was born in Dubuque, Iowa on August 14, 1864 and raised in Utica, New York. He attended Williston Seminary and graduated from Yale University in 1887, where he was a member of Psi Upsilon and Scroll and Key.
He attended Harvard Law School for a year and then continued studying law in Washington, D.C. while he worked as private secretary for William B. Allison, a United States Senator from Iowa. He was admitted to the bar in 1893 and commenced practice in New York City.
Active in politics as a Republican, Sheffield was elected to the New York State Assembly in 1893 and served one term. From 1895 to 1898, he was a member of New York City's Board of Fire Commissioners, and he was the board's president in 1897 and 1898. He was a delegate to every New York State Republican Convention from 1896 to 1924>and a delegate to the Republican National Conventions of 1916, 1920, 1924, and 1936. He was active in judicial reform movements and other efforts to improve local government in New York City.
He was twice offered appointment to the federal bench during the Theodore Roosevelt presidency, both of which he declined. He was also a friend and political ally of Charles Evans Hughes, who was United States Secretary of State from 1921 to 1925. He served as President of The Union League Club from 1921-1924.
In 1924, he was selected to serve as Ambassador to Mexico, with Hughes playing a leading role in obtaining the appointment from President Coolidge. Sheffield served until 1927 and earned notoriety for expressing racist views of and contempt for individuals in the Mexican government. In 1925, he acted on his own to reprimand Mexico's government for its support of Augusto César Sandino's revolutionary movement in Nicaragua, which complicated the Coolidge administration's efforts to maintain a policy of nonconfrontation with Mexico. Sheffield was replaced by Dwight Morrow, who received favorable publicity for his work to create a friendly relationship with President Plutarco Elías Calles.
Sheffield continued to practice law in New York City, and in 1930, he was appointed a Special Ambassador to Venezuela, leading a US delegation for ceremonies dedicating a statue of Henry Clay in Caracas.
Sheffield died in Saranac Lake, New York on September 2, 1938. He was buried at Forest Hill Cemetery in Utica.
In 1898, he married Edith Tod (1873–1956) of Cleveland, Ohio, a great-granddaughter of David Tod. They were the parents of a son, Frederick Sheffield, who participated in the 1924 Summer Olympics as a member of the United States rowing team that won a gold medal; he later worked as an attorney in New York City.