Silvagni did not play in the TAC Cup as a junior, but instead played for his school, Haileybury College winning a premiership in his final year in 2005. He joined the Casey Scorpions in the Victorian Football League in 2006, winning selection in the VFL's under 23 team. He had two successful seasons in 2008 and 2009, playing mainly at full back, but also pinch-hitting in the ruck or forwardline, finishing 5th in 2008 (2nd best non-AFL aligned player) and then third in 2009 in Casey's best and fairest award, being awarded the most consistent award in 2008 and winning selection in the 2009 VFL Team of the year.
Silvagni attended the Victorian state screening session prior to the 2006 AFL Draft, but was not selected by any teams. In 2009, Silvagni moved to Western Australia to train with the Fremantle Football Club, who selected him with their second selection in the 2010 Rookie Draft, the 24th overall. He was one of eight players from the VFL selected in the rookie draft, including Michael Barlow, who was also drafted to Fremantle from the Werribee Football Club.
After playing well in Fremantle's three NAB Cup pre-season games, Silvagni was promoted to Fremantle's senior list and was selected to make his AFL debut in the opening round of the 2010 AFL season.
Silvagni made his debut for Fremantle in the opening round of the 2010 AFL season and performed well, taking 7 marks and gathering 14 possessions, including an impressive 55-metre goal as Fremantle recorded a 56-point victory over the Adelaide Football Club. The celebrations, however, were tempered when Silvagni's former captain at the Casey Scorpions, Kyle Matthews, was knocked unconscious when he was involved in a fight after watching the game at a hotel in Hawthorn.
His drafting as a "mature-aged rookie" and subsequent composed performances in his first few games are seen as an indication of a change to the recruiting philosophy to not only draft the best 18-year-old players, but also to consider older players.
At the conclusion of the 2016 season, he was delisted by Fremantle. He was subsequently drafted by Carlton in the 2017 rookie draft.
Silvagni is the second cousin of Carlton legend and AFL full-back of the century, Stephen Silvagni. Alex's father Eric and Stephen's father Sergio are first cousins. His mother is Anglo-Indian.
Silvagni made his senior debut for the Carlton Football Club in Round 6, against the Sydney Swans. The match was played at the MCG and was a 19 point victory for Carlton. Silvagni played a majority of the game on Lance 'Buddy' Franklin and kept him to one goal.Statistics are correct to the end of the 2016 season