Legien was born in Marienburg, Province of Prussia, (now Malbork, Poland), to Rudolf, a tax official, and Maria Legien. His parents died in his childhood and Legien grew up in an orphanage in Thorn, Province of Prussia (now Toruń) from 1867 to 1875. He became a wood turner and served in the Prussian Army from 1881 to 1884. He joined the Social Democratic Party of Germany (SPD) in 1885, a wood turners’ union in 1886 and worked as a turner in several cities in Germany until 1891, since 1886 in Hamburg.
In 1887 Legien became the first chairman of the German Association of Turners and of the General Commission of the German Trade Unions (Generalkommission der Gewerkschaften Deutschlands) in 1891, a position he would hold until its dissolution in 1919. He was elected a member of the German Parliament in 1893 (until 1898) and again in 1903 (until his death in 1920). He became the leader of the SPD's right wing and opposed its more leftist factions.
He took part in the International Workers Congresses of Paris, 1889.
Legien became Chairman of the International Secretariat of National Trade Union Centres in 1903 and first President of the International Federation of Trade Unions in 1913 until its dissolution in 1919.
In 1912, Legien gave a keynote address at the convention of the Socialist Party of America in Indianapolis which was credited with persuading the convention to reject the anarcho-syndicalist program of Bill Haywood.
At the outbreak of World War I he supported the war with "patriotic fervor" and the SPD-majority’s Burgfriedenspolitik, a "civil truce", which assured the German government not to "obstruct the German war effort". Legien and other leading Social Democrats expected this policy to end the animosity and discrimination of socialist workers in Germany, while the German Empire's government (particularly the War Ministry) evaluated organised labour as an important factor in war industries. As a result, workers became a mobilized, disciplined loyal force in the war effort in return for concessions, and the German labour movement became an obstacle against opposition to war. In the context of the separation of the opposing Social Democratic minority, which led to the foundation of the Independent Social Democratic Party of Germany (USPD), Legien and Gustav Bauer declared that the "Jewish gang" must be dealt with, in attempt to drive them out of their faction. Robert S. Wistrich classifies Carl Legien as belonging to a group of whom some had antisemitic tendencies. During the war he worked in variety of ways to help German war effort. He threw down calls from socialists in USA to mediate an end of the war with the German government, while defending the resumption of submarine warfare by German Kriegsmarine as response to the rejection of "Germany's sincere offer of immediate peace negotiations"
On 15 November 1918 he signed the Stinnes-Legien-Agreement with industrialist Hugo Stinnes, an agreement in which the German employers for the first time accepted nationwide unions as legitimate workers-organisations and which introduced an eight-hour day, workers councils in plants with more than 50 employees and parity employment offices. The employers agreed to stop discrimination of union members and their support of "house unions" (yellow unions) while the unions rejected radical socialists’ demands. Most of the agreement's regulations became part of the Weimar German constitution.
In 1919 he became the first Chairman of the Allgemeiner Deutscher Gewerkschaftsbund. To Legien the expected loss of Upper Silesia to Poland after World War I would intensify the 'psychological' impact of the "draconian peace" on the German working class.
He countered the right-wing Kapp Putsch of March 1920 by organizing a massive general strike in Germany with about 12 million employees following the joint call of the legal government and the unions. The strike immediately halted all production, transportation, mining and public services, it was "the strongest mass movement the German proletariat ever created" and "gave the Kapp régime its death blow".
At that time he declined Friedrich Ebert’s offer to become Chancellor of Germany.
Legien died after a short illness in Berlin and was buried at Zentralfriedhof Friedrichsfelde.
In 1922 the Stinnes-line named "MS Carl Legien" in his honour.
Bruno Taut's "Wohnstadt Carl Legien", a social housing project of the 1920s and part of the UNESCO World Heritage Site Berlin Modernism Housing Estates, bears his name as well as several streets all over Germany. A memorial was erected in Berlin-Kreuzberg.