Leinsdorf was born to a Jewish family in Vienna, and was studying music at a local school by the age of 5. He played the cello and studied composition. In his teens, Leinsdorf worked as a piano accompanist for singers. He studied conducting at the Mozarteum in Salzburg, and later at the University of Vienna and the Vienna Academy of Music. From 1934 to 1937 he worked as an assistant to the noted conductors Bruno Walter and Arturo Toscanini at the Salzburg Festival.
In November 1937, Leinsdorf travelled to the United States to take up a position as assistant conductor at the Metropolitan Opera in New York City. As it turned out, his departure from Austria came a few short months ahead of the Anschluss of March 1938, when the country was taken over by Nazi Germany. With the assistance of freshman Representative from Texas Lyndon B. Johnson, he was able to stay in the United States, and became a naturalized American citizen in 1942.
While at the Met, Leinsdorf was particularly noted for his Wagner performances; after the sudden death of Artur Bodanzky in 1939, he was named the Met's "head of German repertoire". From 1943 he had a brief three-year post as Music Director of the Cleveland Orchestra, but was absent for much of this tenure because he was drafted into the United States Armed Forces for World War II; the orchestra did not renew Leinsdorf's contract. Many years later, in the transition in Cleveland from Lorin Maazel to Christoph von Dohnányi between 1982 and 1984, Leinsdorf returned to lead several concerts; Leinsdorf described his role as "the bridge between the regimes".
Leinsdorf was the principal conductor of the Rochester Philharmonic Orchestra from 1947 to 1955. He came to despair of what he saw as Rochester's insular musical culture, famously remarking that "Rochester is the best disguised dead end in the world!" Subsequently, he was briefly head of the New York City Opera, before resuming his association with the Met. In 1962 he was named music director of the Boston Symphony Orchestra. His time in Boston produced many recordings for RCA, but was also marked by controversy, as he occasionally clashed with musicians and administrators.
On November 22, 1963, during a Boston Symphony concert, Leinsdorf had to announce the reports of President John F. Kennedy's assassination in Dallas, Texas to a shocked audience. He and the orchestra followed the news with a performance of the Funeral March from Beethoven's third symphony. In 1969 Leinsdorf left the Boston post. He would continue to guest-conduct operas and orchestras around the world for the next two decades, being particularly associated with the Metropolitan Opera and the New York Philharmonic. He also served from 1978 to 1980 as principal conductor of the (West) Berlin Radio Symphony Orchestra. He died of cancer in Zürich, Switzerland, at the age of 81.
Leinsdorf is also known for his arrangements of orchestral concert suites of music from major operas. They include: Claude Debussy's Pelléas et Mélisande, Richard Wagner's Parsifal, and Richard Strauss's Die Frau ohne Schatten.
Leinsdorf made numerous recordings throughout his career, including some 78-rpm discs for Columbia Records with the Cleveland Orchestra. He made a number of recordings with the Los Angeles Philharmonic for Capitol and later for Sheffield Labs. In the 1950s, he was conductor for a series of complete stereophonic opera recordings made in Rome, beginning with Puccini's Tosca with Zinka Milanov, Jussi Björling, and Leonard Warren for RCA Victor. He continued to record for RCA Victor as music director of the Boston Symphony. Later he again made additional operatic recordings, including the first complete stereo recording of Erich Wolfgang Korngold's Die tote Stadt, with Carol Neblett and René Kollo. Leinsdorf conducted the BSO with pianist Arthur Rubinstein in pianist's second complete recording of Beethoven's piano concertos, Brahms' First Piano Concerto, and Tchaikovsky's First Piano Concerto.
On video Leinsdorf conducts the Vienna Symphony in Johann Strauß: Famous Works. Available on Silverline Classics in Dolby Digital, 2003.
Leinsdorf with the BSO appeared regularly on local broadcasts from WGBH-TV. On August 17, 1967, Leinsdorf conducted the Boston Symphony Orchestra in a two-hour primetime special telecast in color on NBC, a reflection of the days when a commercial network would periodically broadcast a full-length classical concert. The program, entitled An Evening at Tanglewood, featured violinist Itzhak Perlman as guest soloist.