Frazier played college football at Alcorn State University, where he lined up in the defensive secondary next to Roynell Young.
Frazier was a part of the 1985 Chicago Bears team that won Super Bowl XX. During the 1985 championship season, Frazier led the team with six interceptions. His playing career was cut short due to a knee injury he suffered returning a punt in the second quarter of the Super Bowl. He was also a member of the "G Crew" in the video: The Super Bowl Shuffle.
Frazier started his coaching career in 1988 as the first head coach at Trinity College in Illinois, now known as Trinity International University. He held the position for nine seasons, built the NAIA program from the ground up and won a pair of Northern Illinois Intercollegiate Conference titles before he moved on to the University of Illinois in 1997 as the Illinois defensive backs coach.
In 1999 Frazier joined first time head coach Andy Reid as defensive backs coach for the Philadelphia Eagles. It was there that Frazier worked as a fellow assistant with the Vikings future head coach, Brad Childress. The Eagles defense improved steadily in the four years that Frazier was there.
He was defensive coordinator of the Cincinnati Bengals from 2003 to 2004 where he helped turn the unit into a group that increased takeaways from 24 in 2003 to 36 in 2004. The Bengals' 36 takeaways ranked third in the NFL in 2004. The 2004 Bengals notched 20 interceptions, the most since 1996. The Bengals' defense improved from 28th in the league in total yards allowed in 2003 to 19th in 2004, and declined in the two years following his dismissal.
In 2005, Frazier was hired by Tony Dungy as a defensive assistant for the Indianapolis Colts, receiving the title of special assistant to the head coach as well as defensive backs coach. He was specifically brought in by Dungy to help the Colts' young corps of defensive backs. During his time in Indianapolis the Colts passing defense improved from 15th in 2005 to second in 2006. On February 4, 2007 the Colts beat Frazier's former team, the Chicago Bears, in Super Bowl XLI. Four days later, on February 8, 2007, he became the defensive coordinator for the Minnesota Vikings following the hiring of Vikings defensive coordinator Mike Tomlin by the Pittsburgh Steelers as their head coach.
On November 22, 2010, the Vikings fired head coach Brad Childress after the team started 3–7 and named Frazier the interim head coach for the remainder of the 2010 season. Frazier ended the 2010 season with a loss to the Detroit Lions, putting the Vikings in last place in the NFC North. Frazier's record in 2010 was 3–3.
On January 3, 2011, the Vikings and Frazier reached an agreement making him the permanent head coach. In his first season as head coach, the rebuilding Vikings went 3–13, their worst season since 1984. On December 5, 2011, Vikings owner Zygi Wilf stated Frazier would return in 2012.
In Frazier's second full year as head coach, the Vikings improved to 10–6, making the playoffs. The biggest single-season turn around in Vikings history resulted in Frazier finishing fourth in voting for the NFL Coach of the Year Award and the Vikings exercising their fourth year team option to keep Frazier under contract through 2014.
On December 30, 2013, Frazier was fired as the Vikings Head Coach after a 5–10–1 season.
Frazier was hired by the Tampa Bay Buccaneers to be their defensive coordinator on January 3, 2014. On January 7, 2016, the Tampa Bay Buccaneers announced that they would not pick up Frazier's 2016 option.
On January 15, 2016, Frazier was hired by the Baltimore Ravens as their secondary coach.
On January 12, 2017, Frazier was hired by the Buffalo Bills to be their defensive coordinator.