Schimmelfennig was born in Bromberg (now Bydgoszcz) in the Grand Duchy of Posen, Kingdom of Prussia. He joined the Prussian army and served in both the 29th Infantry Regiment "von Horn" (3rd Rhenish) and the 16th Infantry Regiment "Freiherr von Sparr" (3rd Westphalian), the latter of which was garrisoned in Cologne. In Cologne he became acquainted with some of the more radical German political groups and was an active participant in the 1848 revolution, but was disillusioned by the outcome of the peace treaty that ended the First Schleswig War.
He supported the March Revolution and was a member of the Palatine military commission that led the Palatine uprising. He was twice wounded in the Battle of Rinnthal, rescued, following which he fled to Switzerland. For his involvement in the revolutionary movement, he was tried in absentia and sentenced to death by the Prussian authorities. He remained in exile in Switzerland, where he met fellow expatriate Carl Schurz, and ultimately they fled together to London via Paris. While in London, Schimmelfenning became a part of the German democratic movement, a sectarian group within the Communist League led by Karl Schapper and August Willich that was in opposition to the main body of the Communist League led by Karl Marx and Friedrich Engels.
In 1854, Schimmelfennig emigrated to the United States and afterwards gained employment with the War Department. Here he maintained his association with the Forty-Eighters, a group of military officers in the failed revolutions of 1848 who had fled to the United States; many ended up serving in the Union Army. He was the author of a book on the Crimean War titled The War between Russia and Turkey (Philadelphia, 1854).
After his efforts with Carl Schurz to raise an all-German cavalry regiment failed (due to Schurz's appointment by President Abraham Lincoln to be his Minister to Spain), Schimmelfennig attempted to raise an all-German regiment in Philadelphia. When he fell ill, others strove to take over control of this new regiment but they failed, thanks to the efforts of Schimmelfennig's friends. The regiment, consisting of Philadelphia and Pittsburgh Germans, was called the 1st German Regiment (of Pennsylvania) and would later be designated the 74th Pennsylvania Infantry
At the time of the Civil War, there was strong nativist sentiment in the Union. This prejudice was directed toward the German troops of the XI Corps, who retreated en masse after they were flanked by Stonewall Jackson at Chancellorsville. The mostly German corps took the brunt of the scorn that poured forth from the press. Among the critics was the corps commander Oliver Otis Howard, who sought a scapegoat for his own mistakes. During the battle, Schimmelfennig commanded a brigade in the 3rd Division of the XI Corps.
At the subsequent Battle of Gettysburg, Schimmelfennig commanded a brigade in fellow Forty-Eighter-turned-major general Carl Schurz's 3rd Division of the XI Corps. For a short time, Schimmelfennig took command of the 3rd Division when Schurz briefly commanded the corps. His brigade suffered greatly, mostly due to a high prisoner rate as hundreds of men became confused in the narrow streets of Gettysburg and ended up being captured by oncoming Confederates. It and Colonel Charles Coster's brigade did their best to cover the retreat of the rest of the XI Corps, but they soon became disorganized and fled too. During the retreat through the town, Schimmelfennig briefly hid in a culvert on Baltimore Street, and then stayed for several days in a shed on the Henry and Catherine Garlach property, avoiding capture. (There is a marker outside the Garlach house commemorating this event.) After the battle, he rejoined the corps, much to the joy of the troops who thought he was dead. However, Schimmelfennig's story was seized upon by news writers and presented as another example of German cowardice.
After the Battle of Gettysburg, from mid-July until early August 1863, Schimmelfennig was moved to command a brigade in 1st Division, XI Corps. He and his brigade were reassigned to the Southern District of the Department of the South, in the Carolinas, serving on Folly Island. He commanded the District of Charleston, then part of the X Corps during Sherman's March to the Sea. After being sidelined for some time by a bout with malaria, Schimmelfennig had the honor of accepting Charleston's surrender on February 18, 1865. His headquarters was the Miles Brewton House. During his time of service in the swamps about Charleston, he contracted a virulent form of tuberculosis which ultimately led to his death in Wernersville, Pennsylvania, where he visited a mineral springs sanatorium in an effort to find a cure.
Schimmelfennig is buried in Reading, Pennsylvania, in the Charles Evans Cemetery, not far from fellow Union general David McMurtrie Gregg.