Randolph was born in Riverside Township, New Jersey, and grew up in two communities in New Jersey, Palmyra and the Glendora section of Gloucester Township. He graduated from Triton Regional High School in 1961, as part of the school's first graduating class.
Randolph earned a Bachelor of Science degree from Drexel University in 1966, majoring in economics and basic engineering. At Drexel, he was president of the debate society, vice president of the Student Senate, and a member of the varsity wrestling squad. In 1969, he received his Juris Doctor from the University of Pennsylvania, summa cum laude. Randolph ranked first in his law school class and was managing editor of the Law Review.
Randolph then clerked for 2nd Circuit Judge Henry Friendly, which began a career in law in Washington, D.C., moving between private practice, government, and academia.
He started as the Assistant to the United States Solicitor General for three years, went into private practice briefly, and returned as the Deputy U.S. Solicitor General from 1975 to 1977. He also taught at Georgetown University Law Center from 1974 to 1978. In 1979, Randolph was appointed Special Counsel to the Committee on Standards of Official Conduct (the Ethics Committee) of the United States House of Representatives, remaining in this position until 1980. He then stayed in private practice, becoming a partner at Pepper, Hamilton & Scheetz, until he moved to the bench in 1990. He held a number of positions while in private practice, including Special Assistant Attorney General for the states of New Mexico (1985–90), Utah (1986–1990) and Montana (1983–1990). He also served as a Member of the Advisory Panel of the Federal Courts Study Committee. From 1971 to 1990, Randolph argued 25 times in the United States Supreme Court.
Randolph was nominated by President George H. W. Bush on May 8, 1990, to a seat on the United States Court of Appeals for the District of Columbia Circuit vacated by Judge Spottswood William Robinson III. He was confirmed by the United States Senate on July 13, 1990, and received commission on July 16, 1990. He assumed senior status on November 1, 2008.
From 1993 through 1995 Judge Randolph was a member of the Committee on Codes of Conduct of the Judicial Conference of the United States, and from 1995 to 1998 served as the Committee's chairman.
Al Odah v. United States was the first appeal before the D.C. Circuit challenging the Bush Administration's policies regarding detention of suspected terrorists at Guantanamo Bay. In Al Odah, Judge Randolph wrote for a unanimous panel that the detainees at Guantanamo Bay did not have rights under the United States Constitution. That decision was reversed by the Supreme Court in Rasul v. Bush. The United States Congress subsequently passed the Detainee Treatment Act, which was intended to reverse the effect of the Supreme Court's decision in Rasul.
Judge Randolph also wrote the majority opinion for the D.C. Circuit in Hamdan v. Rumsfeld. Hamdan involved a challenge to the Bush Administration's military commissions to try designated "enemy combatants" at Guantanamo Bay. Judge Randolph held for a unanimous court that the Administration had authority to conduct the commissions and that they were not in violation of the Geneva Conventions. Judge Stephen Williams concurred in the judgment, disagreeing on the latter point. The Supreme Court reversed the D.C. Circuit in Hamdan v. Rumsfeld. Again, the United States Congress passed legislation, this time the Military Commissions Act of 2006, to reverse the effect of the Supreme Court's ruling.
Rasul v. Bush became Boumediene v. Bush when it came again before the D.C. Circuit. Judge Randolph again wrote the majority opinion. In Boumediene the court upheld the Military Commissions Act, which stripped the federal courts of jurisdiction to hear petitions of habeas corpus from aliens detained by the US Military. This time Judge Judith Rogers dissented. The petitioners in Boumediene asked the Supreme Court to reverse Judge Randolph's opinion. The Court denied their petition, but, in an unusual move, later reversed itself and granted certiorari, then reversed.