Born in Romans-sur-Isère in 1773, Hippolyte Charles joined the French army as a volunteer. In 1796, while Napoleon Bonaparte was busy winning his first victories in Italy, Hippolyte Charles, a lieutenant in a Hussar regiment and deputy to General Leclerc, Bonaparte's brother-in-law, first met Josephine in Paris. They began an affair almost immediately, although she was nine years his senior. Hippolyte Charles was a southerner who made up for his short stature with a very handsome face with a dark complexion and long black whiskers and moustache. According to the Duchess of Abrantes, "Charles spoke only puns and was the buffoon," but she added that, "he was what is called a strange boy, he made people laugh, it was impossible to find a funnier man." Charles was attracted to Josephine for her confidence, power and sexual experience. She was known to have had many lovers and was reputedly well practiced in the arts of lovemaking.
On June 24, 1796, Josephine decided to rejoin Napoleon, accompanied by her lover Hippolyte, her brother-in-law Joseph Bonaparte and Colonel Junot. On 13 July, She found Napoleon at the gates of Milan, but she continued her affair with Hippolyte soon afterwards on the way back to Paris. It is rumoured that the two lovers were involved in some illicit business dealings as well. The recently promoted captain, enriched through these dubious business transactions, was able to leave the army.
On March 17, 1798, the two lovers were denounced to Napoleon, sending him into a great rage. However, Josephine was able to soothe him and convince him that the rumours were untrue. In July 1798, when Bonaparte was in Egypt, the infidelities of his wife were once again reported to him. He wrote to his brother Joseph to prepare for divorce. Bonaparte's letter was intercepted by Admiral Nelson, and the loss of the French fleet prevented any correspondence. Learning of Napoleon's landing at Fréjus, Josephine rushed to him to try with all her charm to change his mind about the divorce. She realised that she had to break up with Hippolyte Charles if she was to continue enjoying all the advantages of being married to Napoleon at the height of his glory. She became the first lady of the country and resided at the palais du Luxembourg in Paris.
In November 1804, Hippolyte Charles bought the estate of Cassan from Francois-Denis Courtillier. The source of the funds for this purchase is rumoured to have originated from his dubious business deals with Josephine. In 1808, during the Peninsular War and under the protection of his old Hussar regiment, Hippolyte Charles travelled to Spain. The journey added considerably to his wealth, as he was able to acquire from a Spanish officer and French soldiers looted treasures, much of them previously looted from South America and the Inca Empire. He befriended the Spanish officer, originally from Buenos Aires, after recognising his peculiar secret society handshake.
Hippolyte Charles sold the Cassan estate in 1828 to Jacques-Honoré Recappé, a former notary public and general counsel of the Seine and Oise region. He decided to retire to his native land, where he bought an even more expensive castle in Génissieux in the Drôme. Hippolyte Charles died there in 1837.
In the novel Les Paysans, the author, Honoré de Balzac bases his character the General Comte de Montcornet on Hippolyte Charles. The same character also appears in the book La Muse du département by the same author where Montcornet serves in the Peninsular war.