Ledger belonged to a huguenot family that emigrated to England in the 18th century; he was born at 1 Bucklersbury, City of London, the son of George Ledger, a mercantile broker, and his wife Charlotte, née Warren. After leaving school he went to Peru and in 1836 was a clerk in a British merchant's office at Lima. It was at Lima that Charles rescued a drowning native Manuel Incra Manami, who offered to become his servant.
He became an expert in alpaca wool, and in 1842 began business as a dealer in South American products. In 1847 he was grazing sheep and cattle half-way between Tacna and La Paz, and in 1852 went to Sydney to inquire into the possibility of introducing the alpaca into Australia. He returned to South America and by 1859 had brought several hundred alpacas to Sydney. This was a hazardous and difficult business as the export of alpacas was forbidden. Ledger was paid £15,000 for his alpacas and given a position in charge of them. The attempt to acclimatize alpacas in Australia ended in failure, but Ledger was not at fault.
Ledger returned to South America in 1864 and turned his attention to another problem. The cinchona tree, the bark of which yields quinine, grew in Ecuador, Peru, and Bolivia, but the export of either trees or seeds was prohibited. The trees were being wastefully cut down without being replaced, and there was some danger that they might become extinct. Some seeds and plants had been introduced into Europe and Asia by Hugh Algernon Weddell in 1848, and Sir Clements Markham went later to Peru, and Bolivia, and succeeded in acclimatizing trees in Asia and the Dutch East Indies.
Ledger, however, found a better variety for producing quinine, now known as Cinchona ledgeriana, and in 1865 under great difficulties collected several pounds of seed and sent them to his brother George Ledger in London. For his share in this work Ledger's servant, Manuel, was arrested in Bolivia and so severely beaten that he died. The seed was sent to London where some of it was purchased by the Dutch government. Seeds were also sent to India and Queensland but the trees do not appear to have been grown in Australia. In 1883 Ledger went to Sydney again and in 1884 took a farm 20 miles (30 km) from Goulburn, New South Wales.
Losing his savings in the bank failures of the early 1890s, efforts were made by Sir Clements Markham and others to obtain some provision for Ledger from the Indian and Dutch governments. Initially this was refused, but in 1897 on Ledger's 79th birthday, he received news that the Dutch government had granted him an annuity of £100 a year. He died eight years later in 1905 at 'Mimosa Cottage', Elswick St, Leichhardt, Sydney. He is buried in Rookwood Cemetery, Sydney.
Ledger did a great service to the world, as millions of cinchona trees grown in India and Java sprang originally from the seeds he collected. By 1900 two-thirds of the world's supply of quinine came from Java, and over 40 years later the Ledger types of cinchona were still the best quinine yielders.