The son of James Christopher Lee and the former Meriam Christina Nelson, Lee was born in 1923 and reared in Lake Providence in East Carroll Parish in far northeastern Louisiana. His father was a carpenter with only a third grade education. His mother, however, was college educated and determined that all four of her children would attend college, which they did. In 1943, he graduated from Baptist-affiliated Louisiana College in Pineville, with service thereafter in the United States Army during World War II. He remained thereafter a long-term member of the United States Army Reserve, reaching the rank of Major. In 1948, he graduated from the Louisiana State University Law Center in Baton Rouge and launched his law practice that same year in Bunkie. He was a board member and attorney for Avoyelles Trust and Savings Bank.
In 1956, Lee convinced the Louisiana State Legislature to adopt legislation to establish a city court in Bunkie and Ward 10 in Avoyelles Parish. He was elected the first of only two judges to serve in that court through 2014. In 1978, he was elected to the 12th Judicial District court, based in the parish seat of Marksville. He held that position until retiring from the bench in December 1985. He continued in law partnership with Thomas A. Durham and later with his son, Nelson Managan Lee, Sr. He left the legal profession in the early 1990s to care for his ailing wife, the former Martha Managan (1926-2003), a daughter of William Henry and Frances Williams Managan, Jr. In December 2011, the Bunkie City Council and Mayor Michael Eugene "Mike" Robertson, Sr. (born July 1950), honored Lee for his civic accomplishments and renamed Pecan Street as "Judge Lee Oaks".
James Nelson Lee married Martha Managan of Lake Charles, Louisiana in a double wedding with Miss. Managan's sister, Emma Jean Managan, who married Rev. Elwood Kinser Wylie in the same ceremony at Trinity Baptist Church in Lake Charles on June 14, 1949. James Lee and Martha Managan first met at the Baptist Student Union on the Louisiana State University campus in Baton Rouge, where Miss. Managan was directing a choir.
Judge Lee was active in the Louisiana Baptist Convention in Alexandria, the Southern Baptist Convention in Nashville, Tennessee, Guidestone Financial Services (formerly, the Southern Baptist Annuity Board), the Louisiana Baptist Children's Home in Monroe, and the First Baptist Church of Bunkie, of which he was a long-term deacon and a youth Sunday school teacher. Lee was also affiliated with the Masonic lodge and the Shriners. In 2008, he was honored by the Louisiana Bar Association as a 60-year member of the state bar.
Mayor Mike Robertson, whom Lee once taught in Sunday school, said that the retired judge "liked his politics. He had a quick wit and humor about him that I've never seen in a person in that position." Robertson originally came to Bunkie when transferred by a department store, and Judge Lee earlier had worked for the F. W. Woolworth Company.
Lee died at the age of ninety in Bunkie. Services were held there at the First Baptist Church on August 9, 2014. He is interred alongside his wife and parents at Bayou Rouge Baptist Cemetery in Evergreen in Avoyelles Parish. In addition to his son, Nelson Managan Lee, Judge Lee is survived by a daughter, Adrienne Lee McKay and husband, Dan, of Bunkie; a younger son, Stuart Rogers Lee of Dallas, Texas; a brother, Albert "Al" M. Lee and wife, Dorothy, of Sarasota, Florida; a sister, Joy Christina Lee Black Turner of El Paso, Texas, and six grandchildren.Lee's six grandchildren include: Holly Elizabeth Descant (née McKay), Martha Ann Lee, Managan Lee Johnson (née McKay), Jane Jarrell Lee, Nelson Managan Lee, Jr., and Dan B. McKay III. He was also survived by two great-grandsons, Theodore Isaac Johnson and Henry James Johnson, named in honor of his great-grandfather James Nelson Lee and great, great grandfather, Henry William Managan, Jr.