Pitts was born in Opelika, Alabama, to Edwin Pitts, Sr., a member of the cavalry, and Erma Mills Pitts. Edwin, Sr. died five months after his son's birth. Pitts' mother gave him the nickname "Alabama" to distinguish him from his father, who was born in Georgia. She was remarried to Robert E. Rudd, and the couple had a daughter, Pitts' half-sister, together. They divorced thereafter, and Pitts and his mother moved to Peoria, Illinois, where Erma became a telephone operator. After going to high school for a year, he enlisted in the Naval Academy at fifteen years old. He served for three years before he received an honorable discharge. After his military career, he wound up living in New York City, where he married in 1928. His mother left Illinois to be with her son in New York.
In 1929, Pitts robbed a New York City grocery store with a gun and stole $76.25 (presently, $1,064). He and his accomplice, James Murphy, were arrested as they tried to get away in a cab. Pitts was implicated in five previous robberies, and for his crimes was sentenced to eight to sixteen years in the Sing Sing prison in Ossining, New York.
At Sing Sing, Lewis Lawes began reforming the prison when he became warden in 1919. This reformation brought about the addition of sports teams to the correctional facility. Pitts played for a football team in the prison system, the Black Sheep, during his sentence. He also played basketball and baseball, in which he had a .500 batting average in 21 career games with eight home runs. Lawes had Pitts' sentence end three years early, and he was released in June 1935 after serving five years and two months. Before his release, however, he was noted by the Los Angeles Times in late 1934 as "the most prominent jail-bird athlete in America." He went through a tryout with two professional football teams during his sentence. On May 22, 1935, Pitts signed a contract with the Albany Senators of the International League to play baseball for $200 a month (presently, $3,494 a month).
After his release from prison in June 1935, Pitts' signing with the Albany Senators generated controversy through the media. W. G. Bramham, the president of the National Association of Professional Baseball Leagues, and Charles H. Knapp, the president of the International League, were against the idea of a former convict playing professional baseball. Knapp refused to approve Pitts' contract and Bramham supported the decision. An executive committee of the National Association held a hearing on June 11, 1935, to review Bramham's actions. The committee supported Bramham, and Pitts announced that he would appeal to Commissioner Kenesaw Mountain Landis. On June 18, Landis declared that Pitts could play professional baseball due to the "complete reformation in Pitts' character" since the robberies.
For the 1935 season, Pitts had a batting average of .233 in 116 at bats. Because of injuries, he only played in 43 games for the Senators in 1935.
Pitts signed a one-year contract worth $500 per game (presently, $8,734 per game) on September 9, 1935, with the Philadelphia Eagles of the National Football League (NFL) to play halfback and defensive back. Signed primarily for publicity reasons, he played in three games for the Eagles, recording two receptions for 21 yards. After the fourth game of the season, Pitts was released.
He played basketball in 1936 for a traveling team called the "Alabama Pitts All-Stars." After re-signing with the Senators, he was demoted to the York White Roses of the New York-Pennsylvania League in 1936. The White Roses were forced to move to Trenton, New Jersey, due to flooding, where they became the Trenton Senators on July 2. Pitts finished the season with a .224 batting average in 156 at bats. His last game as a Senator was on July 6.
Pitts signed with the Charlotte Hornets of the "outlaw" Carolina League a few days later. Later in 1936, he played football for the Stapleton Buffaloes and New Rochelle Bulldogs of the American Association.
Pitts signed with the Winston-Salem Twins of the Piedmont League in 1937. During his time with the team, he had a batting average of .278 in 23 games. He was released by the Twins in June 1937. He signed with the Gastonia Spinners of the Carolina League, but was released later the same month. The Valdese Textiles of the Carolina League picked him up, and he had a batting average of .333 with the team.
In between the 1937 and 1938 seasons, he worked in a hosiery mill in Valdese, North Carolina. He spent the 1938 season with the Textiles and Lenoir Finishers, with a batting average of .268. He returned to Valdese to work after the 1938 season. He was hired as the baseball coach at Valdese High School in 1939. In 1940 he signed with the Hickory Rebels of the Tar Heel League, and he had a season batting average of .303.
On June 7, 1941, Pitts was stabbed to death in a tavern in Valdese, North Carolina, when he tried to dance with a woman that another man was dancing with.