Scott, originally from Newark, New Jersey, spent much of his time behind bars starting at the age of 13, when he was sent to a reformatory for truancy. He first took up boxing while serving time at Trenton State Prison and was released in 1968, but only pursued the sport seriously after a subsequent robbery conviction and being incarcerated again.
As part of its rehabilitation system, the New Jersey prison system offered inmates a variety of job training programs—including training in boxing, whether as a fighter, a trainer, or a cutman. Scott became the light heavyweight champion of the New Jersey prison system.
In 1974, Scott was released on parole to pursue a professional boxing career. Fighting out of Miami Beach, Florida, he won 10 bouts and had one draw between January 1974 and February 1975. However, Scott returned to New Jersey where he was convicted of robbery. An additional charge of murder resulted in a hung jury.
From then to the end of his career, all of Scott's professional fights were held in Rahway State Prison. In Scott's third professional bout after returning to prison, he defeated Eddie Gregory, who was then the WBA's No. 1 light heavyweight contender. (Gregory later changed his name to Eddie Mustafa Muhammad and gained the WBA light heavyweight championship in 1980.)
At least four of Scott's professional bouts held in prison were broadcast by NBC Sports, two by CBS Sports and one by HBO. ABC Sports, on the other hand, declined to provide Scott with national television coverage due to his felony conviction and incarceration.
Although Scott was considered a potential challenger for the light heavyweight championship, the WBA removed him from its rankings in October 1979. The WBA cited concerns that as an imprisoned convict, Scott did not set a "good example", and that his opponents were disadvantaged because they had to come to the prison for all of his bouts. Sports Illustrated questioned whether those were the real motives for removing Scott from the rankings, given that the same conditions had applied when the WBA had started to rank him the year before.
As a result of losing his ranking, Scott announced his retirement from boxing. Nevertheless, Scott changed his mind shortly afterward and announced that he was "unretired" and would fulfill his contracts.
On May 25, 1980, Scott had his first professional loss, an upset in which he was defeated by Jerry "Bull" Martin. In January 1981 he was retried in the death of Everett Russ and was found guilty. He was sentenced to life in prison on March 20, 1981. The loss to Martin and subsequent conviction for murder marked the functional end of Scott's consideration as a serious contender for the light heavyweight title.
Scott's last professional fight was on September 5, 1981, when he lost a unanimous decision to Dwight Braxton, later known as Dwight Muhammad Qawi. Qawi himself had formerly been an inmate at Rahway State Prison.
Scott finished his boxing career with a record of 19 wins, 2 losses and 1 draw.
Scott was released from prison in 2005 after serving a 28-year term. In 2012 he was inducted in the New Jersey Boxing Hall of Fame. He currently suffers from dementia and lives in a New Jersey nursing home.