After playing one season of junior college ball at Cerritos College in California, Nelsen moved on to USC, where he was a key player during his first two seasons, leading the squad in total offense in both 1960 and 1961. In 1962, he split time at the position with Pete Beathard, helping the team capture the national championship.
Nelsen was drafted in the 10th round of the 1963 NFL Draft by the Steelers, but saw limited action during his first two seasons. In 1965, new head coach Mike Nixon gave Nelsen the starting job. However, the struggling team finished 2-12, offering him little help, with Nelsen throwing for eight touchdowns, in addition to 17 interceptions. In the November 14, 1965 game against the Dallas Cowboys, Nelsen suffered a knee injury that plagued him the remainder of that year and underwent surgery after the season.
The following season, Nelsen suffered a knee injury against the Detroit Lions in the season's second game. Originally scheduled to be out of the lineup for six weeks, Nelsen would not return until the season's 12th week. The limited time on the field allowed him to set a league record for fewest interceptions (100 minimum passes) with just one on the campaign. In the season finale on December 18, he completed his final 11 passes to defeat the expansion Atlanta Falcons.
In 1967, the injuries continued for Nelsen when during the closing moments of the September 24 game against the St. Louis Cardinals, he was injured following a tackle by Chuck Walker. He missed the next four games, but came off the bench to engineer a 14-10 comeback victory over the expansion New Orleans Saints on October 29.
On May 14, 1968, Nelsen was traded, along with defensive back Jim Bradshaw, to the Browns. In return, the Pittsburgh Steelers received quarterback Dick Shiner, defensive tackle Frank Parker and an undisclosed draft choice. The trade wound up helping both teams, as within a month into the season, both Nelsen and Shiner were starting at quarterback for their new clubs.
After serving as Frank Ryan's backup for three games in 1968, Nelsen was elevated to the starting role, and quickly made his mark with a shocking 30-20 upset of the previously unbeaten Baltimore Colts on October 20. By the end of the season, Nelsen had led the team to the playoffs, winning nine of his 11 starts.
In 1969, Nelsen had another strong season, leading the Browns to a 10-3-1 record and a postseason berth, as well as being selected to his first and only Pro Bowl. One week after throwing for five touchdowns in a 42-10 victory over the previously unbeaten Dallas Cowboys on November 2, Nelsen spent a miserable afternoon against the Minnesota Vikings on the losing end of a 51-3 score. In addition, he briefly left that game with a pinched nerve in his throwing arm, a malady that would trouble him for the rest of the season.
In game two of the 1970 NFL season, Nelsen took yet another hit to his knees in a September 27 game against the San Francisco 49ers. The injury forced him out of the game and kept him on the sidelines the following week as well. After a midseason slump, he was replaced for one game by rookie Mike Phipps, but returned to start the final five games. Unfortunately, the Browns dropped to a 7-7 mark and missed the postseason.
In 1971, Nelsen led the Browns to four wins in their first five games, but another midseason slump once again relegated him to the bench for one game. Unlike the previous year, however, Nelsen led the team to five straight victories to once again reach the playoffs. Nelsen briefly won the starting job to start the 1972 season, but gave way to Phipps after a 26-10 loss to the Green Bay Packers in the opening game. Weeks after the conclusion of the season, Nelsen underwent his fifth knee operation and announced his retirement.
In 2012, the Plain Dealer named Nelsen #52 on its list of the top 100 Browns of all time.
On February 14, 1973, he was hired as an assistant coach for the New England Patriots, but resigned after two years. After briefly considering the Browns' open head coaching slot, he accepted an assistant position with the Atlanta Falcons. He resigned that position following the dismissal of head coach Marion Campbell.
On February 9, 1977, he was named quarterback coach of the Tampa Bay Buccaneers, reuniting him with his former college coach, John McKay. He spent the next six seasons in that role until being fired amid reports of conflict with Buccaneers quarterback Doug Williams and McKay. After sitting out the 1983 NFL season, Nelsen was hired by Detroit Lions' head coach Monte Clark, a former Browns teammate, as their offensive coordinator on January 20, 1984.
Nelsen has since retired from coaching.