Ashe was born in Knoxville, Tennessee, where he attended public school. He attended the Groton School in Groton, Massachusetts and subsequently the Hotchkiss School in Lakeville, Connecticut. He graduated from Yale University in 1967 with a BA in history. At Yale, Ashe was a member of the Skull and Bones society, as was George W. Bush. In 1974 he earned his law degree from the University of Tennessee College of Law. Before becoming an elected official, Ashe worked as an intern for Congressman Bill Brock, and as a staff assistant for Senator Howard Baker.
In 1968 Ashe was elected to the Tennessee House of Representatives; he was only 23 years old at the time. After serving three terms in the State House, Ashe won the August 1974 Republican primary for a Tennessee Senate seat representing Knox County, Tennessee. In a lawsuit brought by a former legislator Ashe had defeated in 1972, the Tennessee Supreme Court ruled Ashe ineligible to be the Republican nominee as he would not meet the minimum age qualification of 30 on the day of the general election in November. The Knox County GOP then nominated his mother, Martha Ashe, to replace him as the nominee. She was elected by the voters with the promise to resign in January 1975 when Ashe turned 30. Upon her resignation the Knox County Commission appointed Victor Ashe to replace her; he was later elected to the position and served for nine years.
From 1967 to 1973, during the Vietnam War, Ashe was a member of the United States Marine Corps Air Reserves. He was also the Executive Director of the Americans Outdoors Commission from 1985 until 1987.
He ran unsuccessfully for the U.S. Senate in 1984 against future Vice President Al Gore. A third candidate in that race, Ed McAteer of Memphis, had moved from Republican to Independent ranks and five years earlier had founded the Christian right Religious Roundtable.
Ashe was elected to be the mayor of Knoxville in November 1987. He served 16 years as mayor, the longest term in the city’s history. As mayor, Ashe led several initiatives to improve Knoxville’s civic and financial footing. These initiatives focused on such things as waterfront development along the Tennessee River and the building of a convention center to attract tourism and business. The latter was a cause for controversy, with many residents saying that the city of Knoxville did not offer enough amenities to attract would-be events or shows to a convention center. Other initiatives included downtown redevelopment and sign and billboard control.
Ashe stressed diversity within his administration, noting when he left office the growth of minorities and women on commissions and boards during his time as mayor. While Ashe often had the support of the city council, he was unable to get it to institute a Police Advisory Review Committee to perform as a civilian review of police action, something which, in 1998, he felt was necessary to investigate the questionable deaths of three black men at the hands of police officers. At the time, hostility in the black community toward the police department was extremely high due to these deaths. Ashe circumvented the council and established the committee by executive order. Three years later, the council unanimously ratified the order, defusing growing protests for recall elections for Ashe and other councilors .
As mayor, Ashe favored preserving buildings that had possible historic value, at one point threatening to put historic zoning restrictions on a building its owners wished to demolish.
Ashe was a strong supporter of parkland in the city, and during his tenure, parkland in Knoxville was increased from 700 to 1,700 acres (7 km2) and 30 miles (48 km) of greenway was added. He initiated a program called "Penny for the Parks" to create a trust fund that would use a one cent tax and federal matching money to commit $250,000 a year for city parks, greenways and historic preservation. After leaving office, Ashe was awarded a Cornelius Amory Pugsley Medal, which was granted by the American Academy for Park and Recreation Administration in recognition of his work on greenways and public parks.
Ashe was sworn in as a US ambassador to Poland in June 2004; during Ashe's tenure as mayor of Knoxville he, acting on the advice of Dr. Marek Pienkowski, helped to establish a sister city relationship with the city of Chełm, Poland, and led two delegations to that city.
One of the issues Ashe engaged in when he was appointed ambassador to Poland was the difficulty Poles have getting work and tourist visas for the United States. After Ashe met with President George W. Bush and Polish President Aleksander Kwaśniewski in Washington in 2005, Bush suggested that the visa rejection rate for Poles be lowered.
Ashe has also noted that Poland has a growing economy that offers many opportunities for US businesses. He is interested in having his own area of Knoxville and East Tennessee take advantage of such opportunities, and in 2005 advised a group of 16 Knoxville-area businesses to put together a trade mission to Poland.
During his ambassadorship it appears that the Polish government cooperated with the CIA in an 'extraordinary rendition' program in which individuals were transshipped to torture centers in other countries.
On March 16, 2009 he was honoured by President of Poland Lech Kaczyński with Commander's Cross with Star of the Order of Merit of the Republic of Poland, for his contribution to Polish-American cooperation. He was decorated on September 25, 2009 by Mariusz Handzlik, undersecretary of state in President's Office.
Ashe serves on the Broadcasting Board of Governors, the body which supervises the federal agencies which broadcast to foreign countries. In a 2013 report by the Office of Inspector General, Ashe was criticized as a board member "whose tactics and personal attacks on colleagues and staff have created an unprofessional and unproductive atmosphere." Ashe was not directly named, but referred to as a "former mayor," a description which fits only him. Ashe called the report “unwarranted, unfair and factually incorrect” and in his defense pointed to his support from labor.
Ashe and his wife, Joan née Plumlee, have two children together. Ashe is the uncle of professional basketball players Mason Plumlee, Miles Plumlee, and college player Marshall Plumlee.