Swanton was born in Forest Hill in south London, the eldest child of a stockbroker. He was a large baby and known as Jim, a diminutive of "Jumbo", from his earliest years. His father was treasurer of Forest Hill cricket club, and Swanton claimed that, whilst still a baby in his pram, he attended a cricket match at which W. G. Grace, then aged 59, scored a century for London County against Forest Hill.
He was educated at Brightlands prep school (later part of Dulwich College Preparatory School) and Cranleigh School. He did not excel at school academically or in sport, and decided to become a journalist on leaving school.
Swanton started at the Amalgamated Press in Fleet Street in 1924. He became a correspondent for the London Evening Standard in 1927, writing on cricket in the summer and rugby in the winter. He wrote about Test cricket for nearly 70 years, from the 2nd Test against Australia at Lord's in 1930 to the 4th Test against New Zealand at the Oval in 1999. He started a parallel career as a broadcaster for the BBC Empire Service in 1934, and commentated on the MCC tour to South Africa in 1938–39, the first overseas tour to receive live BBC coverage.
Swanton formed his own cricket club in 1935, the Arabs, a nomadic club with no home ground. He also played three county cricket matches for Middlesex County Cricket Club in 1937 and 1938, all against university sides. He did not distinguish himself, scoring only 67 runs in 5 first-class innings. He was also president of Sandwich Town Cricket Club; he is labelled as having served the role for "25 years from 1976", although he died in January 2000.
Swanton served in the Bedfordshire Yeomanry in the Second World War. He was in the rank of acting major when wounded and captured by the Japanese in the fall of Singapore, and spent three years as a prisoner of war. His unit spent time in camps along the Burma–Siam railway, and he contracted polio and lost a considerable amount of bodyweight, but his well-thumbed copy of the 1939 Wisden Cricketers' Almanack boosted morale. He later described playing cricket with makeshift equipment and under conditions of extreme privation and the constant threat of brutality in an article, "Cricket under the Japs", for the 1946 edition of Wisden Cricketers' Almanack.
He became cricket correspondent for The Daily Telegraph in 1946, remaining in that post until 1975. He was also editorial director of The Cricketer from 1967 to 1988. He toured Australia as a cricket correspondent seven times between 1946 and 1975, but never visited Pakistan to report on cricket, only reported on one Test match in India, and refused to visit South Africa from 1964 as a protest against apartheid. His writing style was very spare and simple, reporting what happened and why, without the flourishes of Neville Cardus or John Arlott. John Warr once described it as being "halfway between the Ten Commandments and Enid Blyton". His radio commentary provided a contrast to the romanticism of Arlott.
Essentially a traditionalist, he regretted the passing of the gentleman amateur cricketer, and objected to the commercialisation brought about by Kerry Packer's World Series Cricket. He served on the MCC main committee from 1975 to 1984. He never served as President of the Marylebone Cricket Club, but was elected an honorary life vice-president in 1989. He was president of Kent County Cricket Club in 1981, and president of the Cricket Society and the Forty Club.
He was awarded the OBE in 1965 and the CBE in 1994. He published his autobiography, Sort of a Cricket Person, in 1972, and published Cricketers of my Time, a collection of obituaries from The Daily Telegraph, in 1999.
He made his selections as one of the voters for the Wisden Cricketers of the Century in 2000, shortly before he died of heart failure in Canterbury. Some also said he had also selected – while never actually a selector – every post-war amateur England cricket captain. When The Cricketer completed 70 years in 1991, its then-editor Christopher Martin-Jenkins invited Swanton to select a greatest XI from those 70 years. Swanton's 11 included four Australians, four West Indians, two Englishmen, and one Indian. The team was: Jack Hobbs, Sunil Gavaskar, Don Bradman, Gary Sobers, George Headley, Keith Miller, Alan Davidson, Godfrey Evans, Lance Gibbs, Bill O'Reilly, Malcolm Marshall.
His obituaries were fulsome, with Ted Dexter in The Sunday Telegraph saying "He was the standard by which other cricket commentators were judged". He helped Tony Greig in Greig's early cricketing career in England.
Swanton married Ann Marion Carbutt in February 1958. She was daughter of Reymond de Montmorency, housemaster at Eton College, and the widow of a chartered accountant, George Carbutt. They lived near Sandwich, Kent, and both enjoyed golf at Royal St George's nearby. They also had a winter home at Sandy Lane in Barbados. She died in 1998.
Swanton's wartime experience led him to a deep Anglo-Catholic faith, and he considered becoming an Anglican priest. Lord Runcie, former Archbishop of Canterbury, delivered the address at his funeral. A respected biography of Swanton by David Rayvern Allen published shortly after his death revealed many previously unknown facts about his life.
E.W.Swanton is mentioned in episodes 21 and 23 of Monty Python's Flying Circus.