Born in Regina, Nielsen was elected to parliament in late 1957 (Nielsen lost in the 1957 federal election, but the result was controverted and Nielsen won the resulting byelection) and remained an MP without interruption for 30 years. He was a backbench MP during the Diefenbaker government but became prominent during the Conservative Party's long period in Opposition during the 1960s and 1970s joining the shadow cabinet in 1964. In 1978, he ran for the leadership of the newly formed Yukon Progressive Conservative Party as it prepared for the territory's first partisan elections but was defeated by Hilda Watson by one vote.
With the 1979 federal election, the Tories formed government for the first time in over 15 years and Nielsen was appointed Minister of Public Works in the short-lived minority government of Prime Minister Joe Clark. After the Tories were defeated in the 1980 election, he served as Opposition House Leader from 1981 until 1983, and engineered the "Bell Ringing Affair" to protest the Liberal government's omnibus energy bill. The business of the Canadian House of Commons ground to a halt for three weeks because the Opposition refused to respond to the bell summoning Members of Parliament to come to the chamber to vote.
Nielsen served as Leader of the Opposition in 1983 between the resignation of Joe Clark and the election of Brian Mulroney as PC leader, and continued to lead the party in the House until Mulroney won a seat in a by-election, at which point Nielsen returned to his previous position as House Leader.
When Mulroney became prime minister, he made Nielsen his deputy prime minister from 1984 to 1986, and President of the Queen's Privy Council for Canada from 1984 to 1985. Nielsen was effectively the senior Government House Leader in all but name. He also served as Minister of National Defence from 1985 to 1986.
Nielsen has been called "Yukon Erik," (a reference to wrestler Yukon Eric of the 1950s) but he has also been called "Velcro lips" for a tight-lipped reticence during his time in office. The tenaciousness and aggressiveness that made Nielsen a successful Opposition MP made him a liability as a Cabinet minister as he gave the impression of being secretive and disdainful of criticism by the Opposition and the media. His habit of stonewalling questions had the effect of prolonging the shelf life of political scandals in Parliament, and thus hurt the government's reputation. This became most apparent during the Sinclair Stevens conflict-of-interest scandal, in which Mulroney was out of Parliament for two weeks while the opposition barraged Nielsen with questions. Shortly after Mulroney returned in June 1986, he forced both Nielsen and Stevens to resign from cabinet.
Years afterward, Sheila Copps would remark that the sacking of Nielsen made Mulroney "look decisive", when pointing out the importance of a deputy prime minister in protecting the prime minister from political damage during question period.
Nielsen resigned his seat in Parliament in January 1987 when he was given the position of chairman of the National Transportation Agency. He withdrew from the public service in 1992 to become president of Solar Engineering, Hawaii Inc. and Solar Electric Engineering Distributors Canada.
One of Nielsen's brothers was actor Leslie Nielsen. The relationship formed the premise of an HBO mockumentary titled The Canadian Conspiracy, comically alleging a Canadian subversion of the United States through its media. Nielsen was also a nephew of actor Jean Hersholt.
Nielsen wrote a memoir, The House Is Not a Home (1989, ISBN 0-7715-9426-7), noted for its bracing directness both about his colleagues and about his own personal life.
He died at his home in Kelowna, British Columbia on September 4, 2008, from a massive heart attack. On December 15, the government of Yukon renamed the main airport at Whitehorse, the capital of the territory, to Erik Nielsen Whitehorse International Airport in Nielsen's memory.