Frazier was born in Pikeville, Tennessee, the son of Thomas and Margaret (McReynolds) Frazier. In 1867, his family moved to Davidson County, where his father had been appointed to a criminal court judgeship. Frazier attended Franklin College near Nashville, and obtained his Bachelor of Arts degree from the University of Tennessee in 1878. He was admitted to the bar in 1881, and moved to Chattanooga to practice law that same year.
Described by historian Zella Armstrong as "one of the greatest orators who ever lived in the Volunteer State," Frazier quickly rose to prominence in Chattanooga political circles. In 1894, he challenged incumbent Henry Snodgrass for the Democratic Party nomination for the 3rd District seat in the U.S. House of Representatives. While Frazier had wide support, Snodgrass had more support from the delegates at the party's convention, and won the nomination. Snodgrass was defeated by the Republican candidate, Foster V. Brown, in the general election, however, due in part to bitterness from Frazier's supporters.
During the 1900 presidential race, Frazier served as the state's at-large elector for the Democratic candidate, William Jennings Bryan. He canvassed across Tennessee on behalf of Bryan, gaining invaluable statewide political exposure.
In 1902, Frazier sought and obtained the Democratic nomination for governor in the race to succeed Benton McMillin, who was not seeking reelection. His Republican opponent was Judge H. Tyler Campbell, who had won his party's nomination with the support of party boss Representative Walter P. Brownlow. The 1902 gubernatorial campaign is remembered as the last in which the candidates canvassed the state via horse-drawn carriages. In the general election, which was marked by low turnout, Frazier won easily with 98,902 votes to 59,007 for Campbell, and 2,193 for Prohibition candidate R. S. Cheves.
Frazier sought to bring a business-like atmosphere to the state government, and demanded rigid economic frugality from every department. He vetoed any bill that failed to meet his economic standards, including one that would have raised his own salary. The state had no floating debt during his tenure, and paid off over $600,000 of bonded debt.
Frazier considered public education a priority during his first term. One of the first pieces of legislation he signed was the Romine Bill, which stipulated that any unappropriated funds in the treasury be added to the public school fund. In 1904 alone, this generated over $270,000 for the state's public schools.
In 1903, Frazier signed the Adams Law, a pro-temperance measure sponsored by the Anti-Saloon League. The law extended the state's "Four Mile Law"– which banned the sale of liquor within 4 miles (6.4 km) of a school– to all towns with populations of 5,000 or fewer people (the Four Mile Law originally applied only to rural areas). This effectively banned the sale of liquor in all but the state's largest cities.
In February 1904, a coal miners' strike in the Coal Creek Valley threatened to erupt into a full-blown uprising, raising fears of a revival of the Coal Creek War, which had plagued the valley in the previous decade. Frazier travelled to the valley to meet with the miners, however, and managed to defuse the situation. Frazier also signed into law new mine safety standards in response to a string of mining accidents in previous years, most notably the Fraterville Mine disaster of 1902.
In the gubernatorial campaign of 1904, Republicans nominated Winchester mayor Jesse Littleton to challenge Frazier. In September, Frazier and Littleton engaged in a series of fierce debates, primarily over the Four Mile Law, which Littleton argued should extend to the entire state, while Frazier, not wanting to further agitate liquor interests, argued that larger cities should remain exempt. On election day, Frazier won easily, capturing 131,503 votes to 103,409 for Littleton.
On March 9, 1905, a few weeks after the beginning of Frazier's second term, Senator William B. Bate died, prompting a scramble to fill his vacant seat. Former Governor Robert Love Taylor had for years campaigned unsuccessfully for a Senate seat, and believed himself next in line should one become vacant. Frazier, however, also wanted the Senate seat, and while Taylor was still out of state on a speaking tour, quickly convened the General Assembly to have himself elected to the seat. After this election, Frazier resigned to take his seat in the Senate, and the Speaker of the Tennessee Senate, John I. Cox (who had helped engineer Frazier's election), became governor.
Taylor was outraged by Frazier's actions, and suggested that Frazier, Cox, and Senator Edward W. Carmack were conspiring to control the state Democratic Party. To appease Taylor's supporters, the party allowed Taylor to face Carmack in a primary in 1906 (the first Senate primary in state history), which Taylor won. That same year, the party refused to nominate Cox for reelection.
As a senator, Frazier championed states' rights, and opposed most protective tariffs. He also supported a federal income tax amendment and funding for highway construction. He campaigned for a second term in 1911, but state Democrats, irritated by the intraparty divisions created by his first election, refused to nominate him.
After leaving the Senate, Frazier returned to Chattanooga, where he practiced law with his son, James B. Frazier Jr. (1890–1978), in the firm, Frazier and Frazier. He died on March 28, 1937, and is buried in the city's Forest Hills Cemetery.
Frazier's great-grandfather, Samuel Frazier, and grandfather, Abner Frazier, both fought at the Battle of Kings Mountain in 1780. Samuel Frazier was of Scottish descent, while his wife, Rebecca Julian, was of French Huguenot descent. An uncle of James B. Frazier, Dr. Beriah Frazier (1812–1886), served as Mayor of Chattanooga in 1841, and represented Knox County at the East Tennessee Convention of 1861.
Frazier married Louise Douglas Keith in 1883. They had four children: Anne, James Jr., Thomas, and Louise. James B. Frazier Jr. represented the 3rd District in the U.S. House of Representatives from 1949 to 1963.