Rosa was born Karl August Nikolaus Rose in Hamburg, Germany, the son of Ludwig Rose, a Hamburg businessman, and Sophie Becker. His father subsequently took him to Edinburgh. A child prodigy, Rosa toured in Scotland from age 12 to age 16, eventually earning glowing notices. Beginning in 1859, he studied at the Conservatorium at Leipzig (where he met and became lifelong friends with Arthur Sullivan) and, in 1862, in Paris.
In 1863, Rosa was appointed Konzertmeister at Hamburg, where he had occasional opportunities to conduct. Three years later he visited England, appearing as a soloist at the Crystal Palace. He had considerable success as a conductor both in England and the United States. He travelled to America in 1866 as a member of a concert troupe promoted by the Baltimore impresario Hezekiah Linthicum Bateman that also included the Scottish operatic soprano Euphrosyne Parepa. During this tour, on 26 February 1867 in New York City, he married Parepa, who became known as Madame Parepa-Rosa.
In 1869, in collaboration with the Chicago impresario C. D. Hess, the couple formed the Parepa Rosa English Opera Company in New York and toured in America for three seasons, with Parepa as the star and Rosa as the conductor. It brought grand opera to places in America that had never seen any, performing Italian operas in English, which made them more accessible to American audiences. In 1872, the Rosas returned to England and also visited Europe and Egypt. Rosa changed the spelling of his name after he moved to England, where people took "Rose" as a monosyllable.
In 1873 Rosa and his wife started the Carl Rosa Opera Company (the change in name reflecting her pregnancy) with a performance of William Vincent Wallace's Maritana in Manchester on 1 September, and then toured England and Ireland. Rosa's policy was to present operas in English, and that remained the company's practice. Also in 1873, dramatist W. S. Gilbert approached Rosa about producing a short comic opera based on one of Gilbert's Bab Ballads, "Trial by Jury: An Operetta". Parepa was to play the soprano lead, as part of Rosa's planned season of English opera at the Drury Lane Theatre. Parepa died in January 1874, however, and the project was dropped (until, in 1875, a competing manager, Richard D'Oyly Carte, produced Trial by Jury with music by Arthur Sullivan). Rosa later endowed a Parepa-Rosa scholarship at the Royal Academy of Music in London. He married a second time in 1881. With his second wife, Josephine (d. 1927), he had four children.
The company's first London season opened at the Princess's Theatre in September 1875, playing The Marriage of Figaro, with Charles Santley as Figaro and Rose Hersee as Susanna. In 1876, Rosa staged a second London season, which featured the first performance in English of The Flying Dutchman with Santley in the title role. For the next fifteen years, under Rosa's guidance, the company prospered and earned good notices, with provincial tours and London seasons, frequently in conjunction with Augustus Harris at the Drury Lane Theatre. Such was the success of the company that at one point three Carl Rosa touring troupes were set up. Rosa hired Alberto Randegger as the musical director of the company from 1879 to 1885. In 1880, Sir George Grove wrote: "The careful way in which the pieces are put on the stage, the number of rehearsals, the eminence of the performers and the excellence of the performers have begun to bear their legitimate fruit, and the Carl Rosa Opera Company bids fair to become a permanent English institution." In 1892, Rosa's Grand Opera Company gave a command performance of La fille du régiment at Balmoral Castle.
Rosa introduced many works of important opera repertoire to England for the first time, performing some 150 different operas over the years. Besides Santley and Hersee, Minnie Hauk, Joseph Maas, Barton McGuckin and Giulia Warwick were some of the famous singers associated with the company during its early years. Rosa also encouraged and supported new works by English composers. Frederic Hymen Cowen's Pauline (1876), Arthur Goring Thomas's Esmeralda (1883), Alexander Mackenzie's Colomba (1883) and The Troubabour, and Charles Villiers Stanford's The Canterbury Pilgrims (1884) were commissioned by the company. Earlier English operas by Wallace, Balfe and Julius Benedict were also included in the company's repertoire. An obituarist noted, "He had long looked forward to the time when Sir Arthur Sullivan would have undertaken a grand opera, and to the last had hoped to have been able to produce such a work." Shortly before his death, Rosa launched a light opera company that debuted with Robert Planquette's Paul Jones.
Rosa died suddenly in Paris, on 30 April 1889, and was buried in Highgate Cemetery, London. Rosa's place in English music was indicated by the number of leading musicians who attended his funeral, including Sullivan, Stanford, Mackenzie and George Grove, together with Richard D'Oyly Carte, George Edwardes and Augustus Harris.
By the time he died, Rosa had demonstrated that English opera could be an artistic and financial success. In a memorial lecture, the critic Herman Klein said of him, "From an artistic point of view he achieved triumph after triumph; he lifted English opera out of the slough of despond in which it was found in 1875." Both during his life and after his death, his company had much to do with popularizing opera in England, encouraging native composers and training many singers who went on to international careers. His company survived his death and continued to perform opera in English until 1960. A new opera company, using the name, was revived in 1997 under the artistic direction of Peter Mulloy. This company performs Gilbert and Sullivan and other light operas, as well as grand opera.