At Lawrence Free State High School Ballard competed in football, basketball as well as track. In football, Ballard was a 2006 Preseason Prep Star All-American, he earned first team All-State and All-Conference awards as a senior after being honorable mention all-conference as a junior and sophomore. Prior to his senior season was listed amongst the Rivals.com Top 100 and ranked the #4 tight end prospect in the nation by Sports Illustrated.
For his high school career Ballard caught 50 passes for 558 yards and six touchdowns, including 23 receptions for 287 yards and five touchdowns as a senior. He teamed with fellow All-State/All-Conference players Ryan Murphy,[Mikel Ruder, Brian Murphy, and Kyle Weinmaster to lead their team to an 11-1 record and an undefeated conference mark as seniors, the team's lone loss was to Shawnee Mission West during the state championship playoffs.
Ballard also earned two letters in basketball and four letters in track and field where he earned some notoriety for, despite being 6'4" and 270 pounds he was a member of the school record holding 4x100 relay team that qualified for the state championship meet. Over the course of his track career he had personal bests of 6.80 in the 55m dash (indoor), 22.97 in the 200 meter dash, and ran second leg on a 4x100 team that ran a state best 41.89.
Ballard received scholarship offers from schools such as Oklahoma, Florida State, Georgia, UCLA, Kansas and Kansas State. As a junior, he made a verbal commitment to then Kansas football Coach Mark Mangino, but continued to receive interest from other schools. After receiving notice via the internet that Ballard took a recruiting trip to the Georgia it is rumored that Mangino rescinded the Jayhawks scholarship offer. Soon after Ballard switched his verbal commitment to the Iowa.
At the University of Iowa Ballard made an immediate impact by playing as a true freshman and collecting 15 tackles, four tackles-for-loss and 2.5 quarterback sacks from the defensive end position. He also returned two kickoffs for 32 yards on special teams that season. As a sophomore Ballard become a starter at defensive end for the Hawkeyes. For the season he tallied 40 tackles, 3.5 tackle-for-loss and 1.5 sacks. Ballard burst onto the national scene as a junior when he played both defensive end and defensive tackle, starting at both positions throughout the season. He teamed with fellow Hawkeye Adrian Clayborn to become one of the most feared defensive lines in the nation. For the season he totaled 54 tackles, nine tackles-for-loss and 5.5 quarterback sacks. At the end of the season, he was named third Team All-Big Ten by Phil Steele College Football Magazine and Honorable Mention All-Big Ten by the Coaches Association. Prior to his senior season Ballard was named First Team All-Big Ten by Sports Illustrated, despite being the source of constant double-team blocks he would go on to gather 43 tackles, five tackles for loss and 2.5 quarterback sacks. After the season, he again earned Honorable Mention All-Big Ten by the Coaches Association as well as the Associated Press. He also received the Coaches Appreciation Award and earned an invite to the 2011 Under Armour Senior Bowl.
For his career Ballard totaled 152 tackles, 21.5 tackles-for-loss and 12.5 quarterback sacks.
At the 2011 NFL Scouting Combine Ballard, who had received a first round grade from NFL Draft pundit Mel Kiper Jr., graded out as one of the most athletic defensive lineman at that year's combine. However, soon after, it was reported that he and another player, Justin Houston of the University of Georgia, had tested positive for marijuana. This caused some teams to red-flag him with the much-feared "character flaw" label. It is said that this is the reason he slipped to the fourth round of the 2011 NFL Draft despite performing very well at the Combine and receiving such high marks from draft analysts such as Kiper and Todd McShay.
He was rated eleventh among defensive end prospects for the 2011 NFL Draft.
Ballard had been a part of the Vikings defensive line rotation since being drafted in 2011. He recorded his first NFL sack on December 12, 2012 on Sam Bradford in a game against the St. Louis Rams.
On August 19, 2013, coach Leslie Frazier announced that Ballard would be taking time off from the team to deal with personal issues. The next month, in an interview with USA Today, Ballard stated:
"I wasn't really having a good time playing football. It wasn't fun for me. It wasn't a blast for me..."Making that much money – that was fun. But money is still a material thing. You can always make money. You can't make that time that you lose with your friends and your loved ones. Time is something that you can never get back."