Belgrave was educated at Bedford School and Lincoln College, Oxford. During World War I he served in the Imperial Camel Corps, in Sudan, Egypt and Palestine. In 1915 he was a member of the Darfur Expedition, for which he was awarded the Sudan Medal and Clasp. After the war he was seconded to the Egyptian Government to help the frontier districts administration in the Siwa Oasis. He was an Administrative Officer in Tanganyika Territory in 1924-25.
In the early 1920s the British in Bahrain were concerned to secure the political stability of the island. In 1923 Shaikh Isa ibn Ali Al Khalifa, nearly 80 years old, was induced to hand over power to his son Shaikh Hamad and a series of administrative reforms were carried out. After a succession of Political Agents it became apparent that a permanent administrator should be found who would ensure some political continuity. Shaikh Hamad agreed to appoint a Personal Adviser employed by himself and not by the British Government, who would help him to modernise the state.
There were no readily available candidates and the post was advertised in The Times in August 1925. It is not known how many applicants there were, but after interviews Charles Belgrave was appointed with an annual salary of £720 – enough for him to get married on. During the war he had served with the Frontiers Districts Administration Imperial Camel Corps and had spent two years in the oasis of Siwa Oasis. At the time of his appointment he was on leave after two years in the Colonial Service in Tanganyika. He brushed up his Arabic at the School of Oriental and African Studies in London and arrived in Bahrain in March 1926, to remain there until 1957.
In Bahrain, he was commonly referred to as Al Mustashar (Arabic: المستشار), "the Advisor". To his family and friends, he was known as "Carol".
Amongst his many achievements in Bahrain Belgrave was responsible for the establishment of a system of civil and criminal courts, a functioning and well trained police service, general and widely available education, municipal authorities and political support for the exploration for oil. It was largely the energy of Belgrave in support of the search for oil that put Bahrain ahead of other gulf states in being the first to discover oil in 1932.
Belgrave understood the importance of trade and was the driving force behind the creation of the 'Bab Al Bahrain' (Gateway to Bahrain) structure at the entrance to the market area adjacent to the dhow landing jetties,(Now all reclaimed land) Belgrave is widely reported as having been a well known and popular figure regularly seen riding his horse, wearing a toppee hat, and visiting markets and public gathering places to listen to the views and aspirations of Bahrainis. His office, the 'Advisory', remains to this day the home of Bahrain's courts and justice system.
A general strike was called by the people of Bahrain in March 1956 to remove Belgrave as advisor and force him to leave the country. During the strike, which was estimated to have included 30,000 people, 9,000 of them oil workers, at least 11 people were reported killed in riots on 11 March, after an argument broke out at a vegetable market. The riots started at the Bahrain's oil refinery. A few days before the riots broke out the car of Selwyn Lloyd, the British foreign secretary, was attacked with rocks, with the attackers shouting "Down with Belgrave!" British nationals were forced to stay indoors for safety. As a result, Belgrave's powers were reduced significantly, although Shaikh Salman refused to remove him totally. Shaikh Salman also allowed the establishment of the first legal political party, the National Union Committee (NUC; aka "Committee for National Unity"), a step toward democratic elections. The total killed was later reduced to 5. Some American officials said that this was another example of the British trying to hold on to their colonialism. The uprising is said to have been started through the effort of Gamal Abdel Nasser, the ruler of Egypt at the time, going as far as secretly funding merchants of the bazaar, because of the disagreement that he was having with the British, and other western powers, over Israel. The Soviet Union would take advantage over the tensions caused by this spring upheaval and the criticisms of the western powers (Britain, France and the United States), to supply arms to the Nasser government.
Belgrave was the country's economic adviser for thirty years, but he would leave the country in six months after the protests and having his powers diminished. In December 1956, five NUC men were jailed and found guilty for formulating a plot to destroy the royal palace, kill the royal family, and kill Belgrave.
The biggest long-term benefit to the country during Belgrave's tenure has said to have been the establishment, by his wife Marjorie Belgrave, of schools for girls. However, this caused tensions with some traditionalists.
According to authors Philip L. Kohl, Mara Kozelsky, and Nachman Ben-Yehuda in their work Selective Remembrances, Belgrave was "the first westerner to use and advocate the name "Arabian gulf", first in the journal Soat al-Bahrain (Voice of Bahrain) in 1955."