Brown attended Northwest High School where he was a running back and linebacker. He earned first-team All-District honors and was selected to play in the North-South All-Star game.
Considered a three-star recruit by Rivals.com, he was rated as the 75th best outside linebacker prospect of his class. He originally committed to the Cincinnati, but switched his commitment to Louisville.
Brown recorded one tackle in his collegiate debut versus Kentucky and recorded his first tackle for loss against Memphis. Also had one tackle in the bowl win over Southern Mississippi. He Appeared in all 13 games at linebacker and on special teams and totaled 10 tackles in 13 games.
Brown appeared in all 13 games making 11 starts. He set a career-high with 12 tackles in season-opener versus Murray State. Brown recorded first sack of his college career versus Syracuse. Named BIG EAST Defensive Player of the Week on Oct. 31. He finished tied for second on the team with 84 tackles total.
Brown took over from Dexter Heyman as middle linebacker. He had a breakout season as a Junior in 2012. In the season opener against Kentucky he would finish with 6 tackles. Over the next 5 games he would record 13 total tackles, however he did not play in two of those games. In the next game against South Florida Brown would set a career-high with 17 total tackles. Against his home town Cincinnati Bearcats he was a huge factor on defense in an overtime win with 13 total tackles and an interception. For his effort against South Florida and Cincinnati he would win his second Big East player of the week of his career. In that week he recorded 30 total tackles and an interception. In a win against Temple he would have 10 tackles. In losses against Syracuse and Uconn Brown had 10 and 12 tackles, respectively. In the regular season finale, Brown would total 6 tackles in a win against Rutgers. He ended the regular season with 87 total tackles, a career-high. Brown was key in getting Louisville to the Sugar Bowl. Louisville would upset Florida in the Allstate Sugar Bowl. Brown recorded 13 total tackles and was a key part of the upset win.
Brown returned to Louisville for his senior season. He would lead the team in tackles with 98 total tackles, 71 of them being solo, 14 TFL, and also recorded 5 sacks. Brown forced 3 total fumbles and recovered 2 of the 3, one of which he scored his first touchdown on, returning it 48 yards against Memphis. His score would be the winning touchdown as Louisville won 24-17. Louisville ended their season playing Miami in the Russell Athletic Bowl. Brown recorded 7 tackles and 1 sack in this game as the cardinals won 36-9. Brown played in the 2014 East–West Shrine Game and registered 2.5 total tackles.
In Brown's four years at Louisville, he played in 52 games. He recorded 301 total tackles, 178 of them being solo. Also had 7 sacks, 26 TFL, 2 fumble recoveries out of 3 forced; one of which was returned 48 yards for a touchdown. Had 1 interception.
Brown was rated the fourth best inside linebacker prospect, and was projected to be drafted in the third-to-fourth round by NFLDraftScout.com. Brown was drafted by the Buffalo Bills in the third round (73rd overall) of the 2014 NFL Draft.
In his first career start, the 2014 season opener against the Chicago Bears, Brown recovered a fumble, returning it 27 yards. He played in all 16 games, making 14 starts. He finished his rookie season as the Bills' leading tackler with 109.
In a Week 3 win in Miami in 2015, Brown had two interceptions, returning one 43 yards for a touchdown while also contributing seven tackles (one for a loss). He was subsequently awarded NFL Defensive Player of the Week.