Smith was born in London, England, to parents C. J. Smith, a medical doctor, and Sarah Ann (neé Clode). His sister, Beryl Faber (died 1912), was married to Cosmo Hamilton.
Smith was educated at Charterhouse School and St John's College, Cambridge. He settled in South Africa to prospect for gold in 1888–89. While there he developed pneumonia and was wrongly pronounced dead by doctors. He married Isabella Wood in 1896.
As a cricketer, Smith was primarily a right arm fast bowler, though he was also a useful right-hand lower-order batsman and a good slip fielder. He is widely regarded as one of the best bowlers to play the game. His oddly curved bowling run-up, which started from deep mid-off, earned him the nickname "Round the Corner Smith". When he bowled round the wicket his approach was concealed from the batsman by the umpire until he emerged, leading W. G. Grace to comment "it is rather startling when he suddenly appears at the bowling crease." He played for Cambridge University (1882–85) and for Sussex at various times from 1882 to 1892. While in South Africa he captained the Johannesburg English XI. He captained England to victory in his only Test match, against South Africa at Port Elizabeth in 1888-89, taking five wickets for nineteen runs in the first innings. The English team who played were by no means representative of the best players of the time and nobody at the time realised that the match would enter the cricket records as an official Test match.
In 1932, he founded the Hollywood Cricket Club and created a pitch with imported English grass. He attracted fellow expatriates such as David Niven, Laurence Olivier, Nigel Bruce (who served as captain), Leslie Howard and Boris Karloff to the club as well as local American players. Smith's stereotypical Englishness spawned several amusing anecdotes: while fielding at slip for the Hollywood Club, he dropped a difficult catch and ordered his English butler to fetch his spectacles; they were brought on to the field on a silver platter. The next ball looped gently to slip, to present the kind of catch that "a child would take at midnight with no moon." Smith dropped it and, snatching off his lenses, commented, "Damned fool brought my reading glasses." Decades after his cricket career had ended, when he had long been a famous face in films, Smith was spotted in the pavilion on a visit to Lord's. "That man over there seems familiar", remarked one member to another. "Yes", said the second, seemingly oblivious to his Hollywood fame, "Chap called Smith. Used to play for Sussex."
Smith began acting on the London stage in 1895. His first major role was in The Prisoner of Zenda the following year, playing the dual lead roles of king and look-alike. Forty-one years later, he appeared in the most acclaimed film version of the novel, this time as the wise old advisor. When Raymond Massey asked him to help him understand the role of Black Michael, he answered "My dear Ray, in my time I have played every part in The Prisoner of Zenda except Princess Flavia. And I always had trouble with Black Michael!" He made his Broadway debut as early as 1895 in The Notorious Mrs. Ebbsmith. In 1907 he appeared with Marie Doro in The Morals of Marcus, a play Doro later made into a silent film. Smith later appeared in a revival of George Bernard Shaw's Pygmalion in the starring role of Henry Higgins.
Smith appeared in early films for the nascent British film industry, starring in The Bump in 1920 (written by A. A. Milne for the company Minerva Films, which was founded in 1920 by the actor Leslie Howard and his friend and story editor Adrian Brunel). Smith later went to Hollywood where he had a successful career as a character actor playing either officer or gentleman roles. He was also regarded as being the unofficial leader of the British film industry colony in Hollywood, which Sheridan Morley characterised as the Hollywood Raj, a select group of British actors who were seen to be colonising the capital of the film business in the 1930s. Other film stars considered to be "members" of this select group were David Niven (whom Smith treated like a son), Ronald Colman, Rex Harrison, Robert Coote, Basil Rathbone, Nigel Bruce (whose daughter's wedding he had attended as best man), Leslie Howard (whom Smith had known since working with him on early films in London), and Patric Knowles.
Smith became infamous for expecting his fellow countrymen to report for regular duty at his Hollywood Cricket Club. Anyone who refused was known to "incur his displeasure". Fiercely patriotic, Smith became openly critical of the British actors of enlistment age who did not return to fight after the outbreak of World War II in 1939. Smith loved playing on his status as Hollywood's "Englishman in Residence". His bushy eyebrows, beady eyes, handlebar moustache, and height of 6'4" made him one of the most recognisable faces in Hollywood.
Smith starred alongside such screen legends as leading ladies Greta Garbo, Elizabeth Taylor, and Vivien Leigh and the actors Clark Gable, Laurence Olivier, Ronald Colman, Maurice Chevalier, and Gary Cooper. His films include such classics as The Prisoner of Zenda (1937), The Four Feathers (1939), Hitchcock's Rebecca (1940), Dr. Jekyll and Mr. Hyde (1941), And Then There Were None (1945) in which he played General Mandrake, and the 1949 remake of Little Women starring Elizabeth Taylor and Janet Leigh, in which he portrayed the aged grandfather of Laurie Lawrence, (played by a young Peter Lawford), who generously gives a piano to the frail Beth March (played by Margaret O'Brien). He also appeared as the father of Maureen O'Sullivan in Tarzan the Ape Man, the first Tarzan film with Johnny Weissmüller. Smith also played a leading role as the Earl of Dorincourt in David O. Selznick's literature adaption Little Lord Fauntleroy (1936).
He appears in Dennis Wheatley's 1934 thriller, "Such Power is Dangerous", about an attempt to take over Hollywood, under the fictitious name of 'Warren Hastings Rook' (rather than Charles Aubrey Smith). Commander McBragg in the TV cartoon Tennessee Tuxedo and His Tales is a parody of him. The cartoon character also appears in The Simpsons episode "The Seemingly Never-Ending Story".
Smith died from pneumonia in Beverly Hills in 1948, aged 85. His body was cremated and nine months later, in accordance with his wishes, his ashes were returned to his native UK and interred in his mother's grave at St Leonard's churchyard in Hove, Sussex.
Smith has a star on the Hollywood Walk of Fame.
Smith was an officer in the Legion of Frontiersmen.
In 1933, he was on the first board of the Screen Actors Guild.
He was appointed a Commander of the Order of the British Empire (CBE) in 1938 and was knighted by King George VI in 1944 for services to Anglo-American amity.