Smith grew up in the small city of Towanda in Bradford County, Pennsylvania, where he attended Towanda High School, was a member of the Omega Gamma Delta fraternity, and was graduated in 1910. From Towanda, Smith went on to Amherst College where he was a member of the Chi Psi fraternity and was graduated in 1915, with Phi Beta Kappa and Sigma Xi honors. He received his degree in architecture from the University of Pennsylvania after a short interruption for service in World War I.
James was a member of McKim, Mead and White from 1920 (1924 per Dearinger) to 1961 and the architect for Amherst College from 1930 to 1960. He became a full partner in 1929, and was the last surviving partner of MM&W. He primarily designed academic buildings, but his last major work was the National Museum of American History.
Charles Follen McKim was the classicist, and Stanford White the flamboyant designer. Per Baker, William Rutherford Mead was the MM&W partner who "hired and fired", "steered the ship", and spent his time "trying to keep the partners from making damn fools of themselves." In 1883, Mead married Olga Kilyeni (c1850-1936). Mead retired in 1920, around when James joined the firm. In 1936, "Mrs. Olga Kilenyi Mead, widow of the architect, William Rutherford Mead of McKim, Mead White, who died in her apartment in the Sherry-Netherland Hotel on April 10, last, bequeathed her entire estate to the trustees of Amherst College, Amherst, Mass." His widow received all the estate of about $250,000 per NY Times November 27, 1928 article.
The money was used to build the Mead Art Building, which was designed by James. The building was completed in 1949. James was also the architect of the Memorial Field and War Memorial at Amherst, built from 1945–1946 and dedicated in honor of the alumni who died in World Wars I and II.
In 1923, James was awarded the Rome Prize, a prestigious American architectural award made annually by the American Academy in Rome, McKim's favorite legacy. He was a Fellow at the Academy from 1920–23, and trustee from 1933. He served as president of the Academy from 1937 until 1958 and in 1961 he was awarded a medal for his outstanding service to the organization.
In 1951, he was bestowed an honorary Doctor of Humane Letters from Bowdoin College, who cited Smith's work in architecture for Bowdoin in the building of the Union, swimming pool, the Classroom Building. Smith also designed buildings at Union College, Middlebury College, Tufts University, Colgate University, the American University in Beirut, the University of Connecticut, the University of Delaware, and the University of Pennsylvania.
Smith served in World War II as a lieutenant colonel. He was also a trustee of Pratt Institute
"The present-day National Museum of American History began with the 1923 plan for a National Museum of Engineering and Industry. The architectural firm McKim, Mead and White was contracted for the design of the new building on March 16, 1956. James Kellum Smith of McKim, Mead and White served as designer until his death when Walker Cain of the successor firm Steinman, Cain and White replaced Smith. In December 1957 the National Capital Planning Commission approved the site between 12th and 14th Streets on Constitution Avenue, just west of the Museum of Natural History. The site was cleared and ground was broken in August 1958 and construction began on October 5, 1959. It opened to the public on January 23, 1964 to five and a half million visitors in its first year. In 1969 the museum was renamed to the National Museum of History and Technology and in 1980 it was again renamed to the National Museum of American History." from Buildings of the Smithsonian Institution.
James died on February 18, 1961.
James was a member of the American Institute of Architects (AIA) from 1929 and was named a fellow of the American Institute of Architects in 1944. He was also a member and Vice President of the National Institute of Arts and Letters.
James' son James Kellum Smith Jr., who goes by the name J. Kellum Smith Jr., was Secretary of the Rockefeller Foundation and then Vice President and Secretary of the Andrew W. Mellon Foundation.