Cremins attended the All Hallows High School in the Bronx, New York, where he was born to Irish immigrants. In 1966, he entered the University of South Carolina on a basketball scholarship, where he played under coach Frank McGuire. While Cremins was there, the South Carolina team won 61 games, with 17 losses, while Cremins was the starting point guard for three years for the Gamecocks. Cremins, known as "Cakes", was also the captain of South Carolina's 1969–70 team which went 25–3. He graduated from South Carolina in 1970 with a B.S. degree in marketing, before playing professional basketball for one year in Ecuador.
Cremins started his coaching career in 1971 as an assistant coach at Point Park College in Pittsburgh, Pennsylvania. He next returned to South Carolina to become McGuire's assistant coach and to earn a M.S. degree in guidance and counseling in 1972.
At age 27, Cremins became one of the youngest NCAA Division I head coaches in history when he took charge of the basketball team at Appalachian State University. He inherited a program that had only won 22 games since joining Division I five years earlier, and had just come off the worst season in school history at 3-23. In his first year at Appalachian his team had a record of 13–14, but then they accumulated an 87–56 record over the next five seasons, with three Southern Conference regular season championships. The Mountaineers posted a 23–6 record, and received an NCAA Tournament slot in 1979 after sweeping the Southern Conference regular season and tournament titles.
Cremins's performance at Appalachian State gathered him some national attention in the NCAA coaching ranks, including catching the eye of Georgia Tech athletic director Homer Rice. After Rice persuaded him to come to Atlanta, Cremins was hired as the Rambling Wreck's new head basketball coach at the close of the 1981 season, on April 14, 1981.
When Cremins arrived at Georgia Tech, he walked into a situation that was as bad, if not worse, than what he'd inherited at Appalachian State. Georgia Tech had only notched one winning season in the previous 10 years, and had just suffered the worst season in school history—a 4-23 overall record and a winless record in Atlantic Coast Conference play.
Considering the poor state of the program he'd inherited, Cremins engineered a very quick return to respectability. In only his third year in Atlanta, he led the Yellow Jackets to the 1984 National Invitation Tournament—their first postseason berth of any sort in 13 years. A year later, the Yellow Jackets shocked the ACC by winning a share of the regular season title, then winning the conference tournament. They then advanced all the way to the Elite Eight, tallying an overall record of 27-8. In 1990, Cremins's team advanced all the way to the Final Four in the NCAA Tournament, with an overall 28–7 record. The 28 wins are still a school record for wins in a season.
Cremins was three times the ACC "Coach of the Year": In 1983 with the first ever Yellow Jackets' ACC tournament victory, and an overall 13–15 won/loss record; again in 1985, and again in 1996 when his team posted a 24–12 record, won the ACC regular-season championship with a 13–3 record (only the second time in 15 years that a team from North Carolina hadn't won at least a share of the title), and progressed to the NCAA Basketball Tournament's "Sweet 16". Cremins' coaching of the 1990 Yellow Jackets' team earned him the Naismith College Coach of the Year honor.
Cremins had a host of players that went on to have successful National Basketball Association (NBA) careers. First there was Mark Price (the Cleveland Cavaliers) and John Salley (the Detroit Pistons) in the early 1980s, then Duane Ferrell, Tom Hammonds, Dennis Scott, Brian Oliver, Kenny Anderson, Jon Barry, Travis Best, Stephon Marbury, Jason Collier and Matt Harpring.
Cremins was also an assistant coach on the first-ever gold-medal-winning American World University Games team in 1986, assisting the head coach. Lute Olson of the University of Arizona. Cremins also assisted Olson at the 1986 FIBA World Championship, also winning the gold medal there. During the summer of 1989 he coached the American team that qualified for the World Championships in 1990.
Cremins also assisted the former National Basketball Association coach Lenny Wilkens in the American basketball team's appearance in the Summer Olympic Games of 1996 in Atlanta. This team was the second of the "Dream Teams" in the Olympic Games, and it featured such NBA stars as Charles Barkley, Karl Malone, Reggie Miller, Shaquille O'Neil, Scottie Pippen, David Robinson and John Stockton, several of whom were returning for their second Olympic Games basketball tournament. This "Dream Team" was undefeated in the Olympic basketball tournament, of course, and it defeated the second-place Yugoslavian team 95–69 in the championship game in winning the gold medal.
On March 24, 1993, Cremins agreed to coach basketball at his alma mater, the University of South Carolina, before changing his mind and deciding three days later to continue at Georgia Tech. In 2003, Georgia Tech officially named the basketball court at the Alexander Memorial Coliseum on the Georgia Tech campus, the "Cremins Court". Paul Hewitt would take his place at Georgia Tech in 2000.
Cremins announced his retirement after the 1999–2000 season with a 25-year coaching record of 452–303 (a winning percentage of .599), and with a Georgia Tech coaching record of 354-237 in 19 seasons (also a .599 winning percentage). He is far and away the winningest coach in Georgia Tech history.
With his platinum blond hair, Cremins was an iconic figure at Georgia Tech. It was very common for fans to show up at Alexander Memorial Coliseum wearing blond wigs.
Turning down numerous offers to coach during his retirement, and even an occasional athletic directors job, Cremins toured the country doing motivational speaking, did television commentary on ACC and NCAA basketball, and worked with charities, mainly for Coaches vs. Cancer and the Jimmy V Foundation. Cremins also raised money for a five-to-six week summer program, half of which include disadvantaged children, the Hilton Head Basketball Camp 101.
In 2006, Cremins returned to coaching at the College of Charleston, hoping to restore the basketball program there to the mid-major power status that it experienced under coach John Kresse from 1980 to 2002. To some degree, Cremins succeeded, leading the team to four twenty-win seasons in his six seasons there, in addition to signing the highest-rated high school recruit in school history, Adjehi Baru. Another highlight of Cremins' tenure was a home upset over a North Carolina team ranked number nine in the country on January 4, 2010.
Having taken a medical leave of absence on January 27, 2012 which lasted for the duration of the 2011–2012 season, Cremins retired from coaching on March 19, 2012, citing physical exhaustion.
*= Team record at the time of Cremins' medical leave of absence on Jan. 27, 2012. CofC finished season 19-12 overall with interim head coach Mark Byington being credited with a 7-4 coaching record.
With his wife Carolyn, Cremins has one child, Bobby III, who is married to Jennifer, and two grandsons, Risen (Robert Joseph Cremins IV) and Xander (Alexander Charles Cremins). He also has two step-daughters, Liz and Suzie, from Carolyn's earlier marriage. He lives in Hilton Head, South Carolina.