He was the son of a hotel owner. His first drawing lessons were with a local sculptor named Augustin-Phidias Cadet de Beaupré (1800-?) at the Académie de Lille; then took up painting with François Souchon, a student of Jacques Louis David. He went to Paris in 1853, where he adopted the name "Carolus-Duran". In 1859, he had his first exhibition at the Salon. That same year, he began attending the Académie Suisse, where he studied until 1861. One of his early influences was the Realism of Gustave Courbet.
From 1862 to 1866, he travelled to Rome and Spain, thanks to a scholarship granted by his hometown. During that time, he moved away from Courbet's style and became more interested in Diego Vélazquez. Upon returning to France, he was awarded his first gold medal at the Salon.
In 1867, he became one of the nine members of the "Société Japonaise du Jinglar" (a type of wine); a group that included Henri Fantin-Latour, Félix Bracquemond and Marc-Louis Solon. They would meet once a month in Sèvres for a dinner "à la Japonaise".
He married Pauline Croizette, a pastellist and miniaturist who had posed for his painting "The Lady in Gloves" in 1869. They had three children. Their eldest daughter, Marie-Anne, married the playwright Georges Feydeau.
After 1870, he devoted himself almost entirely to portraits. His success allowed him to open a studio on the Boulevard du Montparnasse, where he also gave painting lessons. He was named a Knight in the Légion d'honneur in 1872; being promoted to Officer in 1878, Commander in 1889 and Grand Officer in 1900.
In 1889 and 1900 he served on the juries at the Expositions Universelles. In 1890, he was one of the co-founders of the second Société Nationale des Beaux-Arts and he was elected a member of the Académie des Beaux-Arts in 1904. The following year, he was appointed Director of the French Academy in Rome, a position he held until 1913.
He was a frequent visitor to the resort at Fréjus, where he owned a small villa. Following his death, the resort named a plaza and a beach after him.
They include John Singer Sargent, Ralph Wormeley Curtis, Kenyon Cox Theodore Robinson, Mariquita Jenny Moberly. Mariette Leslie Cotton, Maximilien Luce, James Carroll Beckwith, Will Hicok Low, Paul Helleu, Alexandre Jean Baptiste Brun , Robert Alan Mowbray Stevenson and Ernest Ange Duez. Of his twenty-five most notable students, the majority were English or American.