Herdman was born in Dunedin. He studied at Otago Boys' High School, and then gained a law degree while working part-time. He was admitted to the bar in 1894, and established a practice in Naseby the following year. He also worked in Palmerston, where he joined the local Freemason lodge. He would retain his connection with the Freemasons over his career, eventually grand master of the Grand Lodge of New Zealand.
Herdman began a political career in Naseby, being elected mayor in 1898. He eventually decided to abandon this by moving to Wellington in 1902, but shortly after he arrived, he was invited to return and stand as a parliamentary candidate in Mount Ida, the Otago electorate which encompassed Naseby. In the 1902 election, Herdman ran on a strongly anti-government platform, harshly criticising the governing Liberal Party. He was elected, and joined the unorganised group of independents who opposed the Liberals. He did not, however, move back to Naseby, instead representing his seat as an absentee.
In the 1905 election, Herdman was defeated. In the 1908 election, he contested the seat of Wellington North, and was elected. The following year, William Massey organised the opposition into the Reform Party, which Herdman became part of.
One of Herdman's early concerns in Parliament was the reform of the public service. Herdman believed that the service was poorly organised and subject to political patronage, particularly under the government of Richard Seddon. In the 1911 election, the Reform Party won office as the Reform Government, and Herdman was able to push through his reform proposals — the Public Service Act of 1912 established uniform conditions of appointment and promotion, and established a supervisory commissioner.
In Cabinet, Herdman served as Attorney-General (10 July 1912 – 4 February 1918), Minister of Justice (10 July 1912 – 12 August 1915), and Minister of Stamp Duties (13 July 1912 – 12 August 1915). In these roles, he gained a reputation as a hard-liner, being described by a contemporary as "ready to employ force ruthlessly for the purpose of upholding law and order". Herdman is believed to have had a major role in the suppression of the Waihi miners' strike, and of the waterfront strike the following year. Both were criticised by many left-wing groups as heavy-handed and repressive, but were defended by the government as necessary steps to preserve order. During World War I, Herdman supported strong measures against anyone protesting New Zealand's participation. As Minister responsible for Police, Herdman also responded harshly to attempts by police officers to form a union, prohibiting the move and attempting to drive the instigators out of the force.
In addition to his political ambitions, Herdman was also interested in becoming a judge of the (original) Supreme Court. As Attorney-General, he had powers to appoint judges, and in 1918, when a position became vacant, he appointed himself. This move was criticised by many as self-interested, especially as Herdman's career as a lawyer had not been particularly distinguished. He served as a judge both in Christchurch and Auckland, and briefly acted as Chief Justice in 1929.
In 1935, he resigned from his judicial position to seek re-election to Parliament, contesting the Auckland seat of Parnell. He was officially an independent, although he had close links to the Democrat Party. He was unsuccessful, and subsequently retired to the Lake Okataina area. He died in Rotorua on 13 June 1953.