Born to a Scottish mother – his English father died when he was only six – he was given a boarder's education at a Scottish school on the banks of Loch Rannoch. He was first educated at Emanuel School in London, before moving to Rannoch School. His rugby skills were honed by the school's science teacher. Smith stated: “The things I learnt while playing rugby at Rannoch were work ethic and fitness. Our pitch was covered in snow and frozen solid for three months of the year, so there was a lot of running up and down hills. We were pretty well drilled and were made to work hard. There were times when it was pretty tough and cold out here but at the end of the day rugby is a hard game and you need to be tough to play it.”
Smith's career spanned the amateur and professional game. He started out at the amateur clubs Dundee High School FP and then played for Watsonians prior to the professionalism of the old Scottish District teams. Of these pre-professional times, Smith was to say: "You find out about survival the hard way. When I joined my first senior club in Dundee, there was an old prop called Danny Herrington, a bit of a local legend, who basically shoved my head up my arse in training, twice a week every week for what seemed like years. Now, that's what you call a learning curve. Those training sessions were my classroom."
On professionalism in Scotland in 1996 he joined Caledonia Reds. The Reds won the Scottish Inter-District Championship in the 1996-97 season, the first Inter-District championship of Scotland's professional era. He played in the Heineken Cup with Caledonia Reds and also played in the 1997-98 Scottish Inter-District Championship the following season.
In 1998, the Scottish Rugby Union decided it could not afford to run the traditional four Scottish districts at a professional level. Caledonia Reds and the Border Reivers were disbanded and the districts merged with Glasgow and Edinburgh respectively. Smith was one of the lucky players that moved to the renamed Glasgow Caledonians, now Glasgow Warriors. He played in the 1998–99 Scottish Inter-District Championship - then a Tri-Series between Edinburgh and Glasgow - and played 4 matches in the European Conference for Glasgow in that season.
After one season at Glasgow Warriors he was taken on at French club CA Brive. He stayed a further two seasons with the club.
Smith returned to the UK in 2001 to join Northampton Saints in the English Premiership making his Northampton debut on 1 September 2001. Over the subsequent eight seasons his appetite for the action never diminished and neither did his popularity with the Saints' fans. His time at Franklin's Gardens included two Powergen Cup finals, a Heineken Cup semi-final and three Heineken Cup quarter finals. He remained at Northampton until his retirement from playing in 2009.
Smith earned his first Scotland cap in 1997 in the Calcutta Cup match versus England at Twickenham. Despite only having a 3 caps to his name at the time he was included by coach Ian McGeechan in the squad for the 1997 British and Irish Lions tour to South Africa. Surprising many, Smith was selected to start all three test matches alongside Paul Wallace and Keith Wood over the other touring props Jason Leonard and Graham Rowntree. His strong scrummaging and good hands alongside many other great team performances ensured that the Lions won over their much more highly fancied hosts and won the series 2–1.
He was also selected in the 2001 Lions tour of Australia and played all three test matches of that series also to become the only Scot to play the six consecutive Lions tests matches of 1997 and 2001 when he played against Australia.
For the next eight-years Smith was to be a first choice starter for Scotland and talismanic figure for the team winning the Five Nations in 1999 and captaining the side throughout the 2001 Autumn Internationals. Smith was also Scotland's Player of the Season in 2000–01. He was included in the Zurich World XV for 2002, a notional team list that was compiled based on ratings from detailed video analysis of performances during the past year. After an international career that included two World Cups and six consecutive Lions tests, Smith's final match in the blue jersey came, appropriately enough, against England in the 2005 Six Nations clash at Twickenham.
Smith was a fan favourite with the Murrayfield faithful with many considering him the greatest loose head prop the team had ever had alongside Grand Slam winning captain David Sole.
Having retired from playing in 2009 he took on the roll of forwards coach at Edinburgh under Rob Moffat.
In May 2012 it was announced that he had been appointed as Forwards Coach with French club Lyon.