A native of Lubbock, Brewer prepped at Lubbock High School where he won All-District, All-State and All-American honors. He quarterbacked the Lubbock Westerners to a 13–0 season and the 4A state championship in 1951. In 1951, he was named the Outstanding High School Football Player in Texas and then played in numerous all-star games including the Oil Bowl, the Greenbelt Bowl and the Texas High School Coaches Association All-Star Game.
Both of Brewer's older brothers, Bob and George, played college football as well. Bob was a quarterback at Texas Tech and George was a running back at Oklahoma, where he played with Darrell Royal. Bob left Tech after his freshman year to fight in World War II. His plane was shot down and he was captured by Nazi sympathizers and disappeared. His body was found buried in the Swiss Alps in 1948 and then returned to the United States.
After earning the starting job on the Longhorns freshman team in 1952, and leading them to three late season victories, Brewer had a chance to compete for the starting job of the varsity in his sophomore year. In the 2nd game of the season, he carried the ball twice for two touchdowns and returned a punt 59 yards to set up a touchdown. After backing up John "Bunny" Andrews, who led the team to a 2-2 start that dropped them from #11 to unranked, Brewer was promoted to the starting role against Arkansas. Brewer then led the team to wins in 5 out of their next 6 games, including upsets of #11 SMU and #3 Baylor, and a share of the Southwest Conference Championship. He led the team that season in passing, kick-off return yards and punt return yards. Against Baylor, Brewer threw the game-winning touchdown pass and against Texas A&M, perhaps Brewer's best game, Brewer set up or scored all three touchdowns. Their only loss that season with Brewer as the starter was the result of a last minute touchdown by co-champion Rice.
Brewer remained the starter in the 1954 season, when the Longhorns started the season ranked #4, but struggled to a 4-5-1 record. Texas lost games to #1 Oklahoma, #2 Notre Dame, #12 Arkansas and #20 Baylor. It managed to tie #12 SMU, mostly thanks to the heroics of back-up quarterback Glen Dyer. Brewer ended the season with another spectacular game against Texas A&M, the first for Texas in the Paul "Bear" Bryant era, throwing for 121 yards - his career high - and 2 touchdowns to win 22-13. The other notable game that season was against Washington State, when Carl Talmadge “Duke” Washington became the first African-American to play at Memorial Stadium.
In spring training in 1955, Walter Fondren outshone Brewer and eventually won the starting position for the opener. Fondren was ineffective in that game, leading coach Ed Price to rotate through his quarterbacks until Brewer, Joe Clements and Dick Miller had all taken the field with similar results, before returning to Fondren. The next week Clements replaced Fondren, had a career game and was promoted to starter for the rest of the season. Brewer played in every game that year - one of the worst in Longhorn history, on both sides of the ball and returning kickoffs, but he never regained the role of starting quarterback.
Brewer graduated with a record of 7-4-1 as a starter and set the school record for highest gain per pass completion.UT - Highest Average Gain Per Pass Completion, Career (15.7 yards), surpassed by Randy McEachern in 1978
He later moved to the Dallas suburb of Richardson and started a 43-year career in banking. In 1998, Brewer was inducted into the Texas Football Hall of Fame in Waco.
Charles' son Robert Brewer became the starting quarterback at Texas in 1981-82. Charles beat Bear Bryant's first two Texas A&M teams in 1954 and 1955 and Robert beat Bryant's second-to-last Alabama Team in 1982, perhaps the only father-son duo to go 3-0 versus Bryant. Brewer's grandson Michael Brewer was the starting quarterback at Virginia Tech in 2014.