Small played for the Toronto Blue Jays, Florida Marlins, Oakland Athletics, Arizona Diamondbacks, New York Yankees and Atlanta Braves. Regarded as a career minor leaguer, he played for a total of more than 20 teams, including his major and minor league stops, during his professional career. He also played for the Edmonton Trappers in 1996 where he threw a no-hitter on August 8.
Small attended South Hills High School in West Covina, California. Small was a 1st-team all-conference selection in baseball, and lettered in basketball. He graduated in 1989.
Small's high school baseball teammates included future major league players Jason Giambi, who he would later play with on the Athletics and Yankees, and Cory Lidle, as well as Jeremy Giambi. Small also played high school baseball with Shawn Wooten, who was on the 2002 World Series Angels team.
Small was drafted by the Toronto Blue Jays in the 22nd round of the 1989 Major League Baseball Draft. He debuted for the Blue Jays in 1994, but was traded to the Florida Marlins in 1995 for minor leaguer Ernie Delgado. Over the next ten years, Small would split most of his time between Triple-A and the majors, with major league appearances for the Oakland Athletics, Arizona Diamondbacks, Atlanta Braves, and a second stint with the Marlins, before joining the Yankees.
In addition, Small was also a member of the Milwaukee Brewers, Tampa Bay Devil Rays, Colorado Rockies and Anaheim Angels organizations, though he never pitched in the major leagues for those teams.
Small began 2005 in the Yankees minor league system, pitching for Double-A Trenton and Triple-A Columbus. Due to injuries, he was called up to the Yankees to fill in, making his first start on July 20, which he won.
Small recorded 10 major league victories without a loss. Small recorded his first major league complete game shutout, which came against the Oakland Athletics on September 3, 2005. His pitch count during the game was just over 110 and he struck out the last batter he faced, Dan Johnson, and Small received the baseball from catcher Jorge Posada as a remembrance.
Small was the first Yankees pitcher to win his first 9 decisions since Tommy John in 1979, and became just the fourth pitcher in history to win at least 10 games without a loss, joining Tom Zachary, Dennis Lamp, and Howie Krist. He finished the 2005 season 10–0 with a 3.20 earned run average and 37 strikeouts. He was re-signed to a one-year, $1.2 million contract during the off-season.
During 2006 spring training, Small injured his right hamstring. He missed the start of the season, but returned to the team on April 30, 2006.
On June 17, 2006, Small was designated for assignment by the Yankees and was outrighted to Triple-A Columbus.
On January 23, 2007, Small signed a minor league contract with the Seattle Mariners. However, on May 18, 2007, he announced his retirement after being released by the Mariners.
On August 2, 2008, Small took part in the 62nd Annual Old-Timers' Day at Yankee Stadium. Six weeks prior, Small survived a bout with encephalitis that included a medically induced coma lasting eight days. He was again honored by the New York Yankees at Old Timers' Day on June 26, 2011, and July 1, 2012.
Small, along with his wife Macy, is a devout Christian. He is active in his church, Fairview Baptist Tabernacle in Sweetwater, Tennessee, where he and his wife are involved in ministry to students.