He was born in 1926 in Grimari, Ouaka Prefecture in the Oubangi-Chari French colony, which is now the Central African Republic. He was a qualified medical doctor and member of the medical faculty in Bangui.
While the country was still a French colony, Goumba was Vice-President of the Government Council from May 1957 to July 1958, President of the Government Council from July 1958 to December 1958, and was briefly Prime Minister in an acting capacity in April 1959, following the death of Barthélemy Boganda in a plane crash. He was defeated in a political power struggle by David Dacko in 1959 and then became a minor opposition party leader. He was in exile in France from 1960 until 1980. He worked for the World Health Organization in Rwanda and then Benin during the 1970s; while in Rwanda, he met his wife, Anne-Marie. Even after his return to the Central African Republic, he was occasionally arrested for political activity. He feuded with all of Central African Republic's presidents until 2003 and was declared by them to be a national traitor.
Goumba has a reputation for honesty and integrity, unusual for a Central African politician. He has stressed the importance of governing without corruption. In the 1981 presidential election, which was won by Dacko (who was nevertheless ousted only a few months later), Goumba took less than 2% of the vote, but in the 1993 presidential election he achieved his best result, coming in second place but being defeated by Ange-Felix Patassé in a run-off, in which Goumba took about 46% of the vote. In 1999 he did poorly by comparison, taking only about 6% of the vote and placing fourth, behind Patassé, André Kolingba, and Dacko.
After Bozizé seized power on March 15, 2003, ousting Patassé, he appointed Goumba as Prime Minister on March 23. His government was formed on March 31, 2003; in its composition it was viewed as a compromise between Bozizé and Goumba, with a number of military allies and relatives of Bozizé receiving key posts while other posts went to associates and allies of various political leaders and to independent figures regarded as competent. The National Transitional Council (CNT) rejected Goumba's proposed programme of general policy on November 5, 2003, saying that the government's objectives, along with the methods of implementing those objectives, were not sufficiently defined in the programme. He had planned to submit a revised programme on December 12, 2003, but on December 11, Bozizé dismissed him as Prime Minister. On the next day Célestin Gaombalet was named to replace him; Goumba was appointed as Vice-President instead.
He was a presidential candidate for the fourth time in the election held on March 13, 2005. Goumba was not expected to win; he received sixth place and 2.51% of the vote. He was one of the five candidates initially approved by the transitional constitutional court on December 30, 2004; seven other candidates were excluded, although six of them were later allowed to run.
On March 14, 2005, the day after the election, members of the Collective of Political Parties of the Opposition (CPPO), including Goumba, signed a petition in which they alleged that fraud had occurred. On March 15, before the election results became available, Bozizé dismissed Goumba from the Vice-Presidency and the position was abolished. According to presidential spokesman Alain-George Ngatoua, this was because the constitution adopted in December 2004 did not provide for a Vice-President, and the dismissal was unrelated to the quality of Goumba's work; Ngatoua said that Bozizé thanked Goumba for facilitating the transitional process through his "wisdom and courage". Goumba expressed disgust at the manner of his dismissal; he said that he had received no notification of the dismissal and found out about it when it was reported on state radio. Goumba's view was that transitional institutions, including the Vice-Presidency, were supposed to be maintained until the installation of an elected government.
Goumba ran for a seat from Kouango in the 2005 parliamentary election, held concurrently with the presidential election, but was defeated; his wife Anne-Marie won a seat, however.
Goumba's son Alexandre was elected to succeed him as President of his party, Patriotic Front for Progress (FPP), on March 5, 2006, after the elder Goumba was appointed to the official post of Ombudsman. As Ombudsman, he called for the government to negotiate with a rebel group after it captured Birao on October 30, 2006.
He presented the first volume of his memoirs, covering the period from 1956 to 1959, on January 14, 2007.
Goumba died at age 82 on 11 May 2009 at a clinic in Bangui after being taken to hospital the previous evening.