After the death of his father, power was not initially divided to include Grifo, another of Charles's sons, by his second wife Swanachild. This was per Charles' wishes, though Grifo demanded a portion of the realm from his brothers, who refused him. By 742, Carloman and Pepin had ousted their half-brother, Grifo, and forced him into a monastery, and each turned his attention towards his own area of influence as majordomo, Pippin in the West (in what was called Neustria, roughly the Area between Nancy and Reims) and Carloman in the East (in what was called Austrasia, roughly what is now is the area between Bruges, Metz and Fulda), which was the Carolingian base of power.
With Grifo contained, the two mayors, who had not yet proved themselves in battle in defence of the realm as their father had, on the initiative of Carloman, installed the Merovingian Childeric III as king (743), even though Martel had left the throne vacant since the death of Theuderic IV in 737.
Unlike most medieval instances of fraternal power sharing, Carloman and Pepin for seven years seemed at least willing to work together; certainly, they undertook many military actions together. Carloman joined Pepin against Hunald of Aquitaine's rising in 742 and again in 745. Pepin assisted Carloman against the Saxons in 742–43, when Duke Theoderic was forced to come to terms, and against Odilo, Duke of Bavaria, in 742 and again in 744, when peace was established between the brothers and their brother-in-law, for Odilo had married their sister Hiltrude.
In his realm, Carloman strengthened his authority in part via his support of the Anglo-Saxon missionary Winfrid (later Saint Boniface), the so-called "Apostle of the Germans," whom he charged with restructuring the church in Austrasia. This was in part the continuation of a policy begun under his grandfather, Pepin of Herstal, and continued under his father, Charles Martel, who erected four dioceses in Bavaria (Salzburg, Regensburg, Freising, and Passau) and gave them Boniface as archbishop and metropolitan over all Germany east of the Rhine, with his seat at Mainz. Boniface had been under Charles Martel's protection from 723 on; indeed the saint himself explained to his old friend, Daniel of Winchester, that without it he could neither administer his church, defend his clergy, nor prevent idolatry.
Carloman was instrumental in convening the Concilium Germanicum in 742, the first major synod of the Catholic Church to be held in the eastern regions of the Frankish kingdom. Chaired jointly by him and Boniface, the synod ruled that priests were not allowed to bear arms or to host females in their houses and that it was one of their primary tasks to eradicate pagan beliefs. His father had frequently confiscated church property to reward his followers and to pay for the standing army that had brought him victory at Tours (a policy supported by Boniface as necessary to defend Christianity). By 742 the Carolingians were wealthy enough to pay their military retainers and support the Church. For Carloman, a deeply religious man, it was a duty of love; for Pippin a practical duty. Both saw the necessity of strengthening the ties between their house and the Church. Carloman donated the land for one of Boniface's most important foundations, the monastery of Fulda.
Despite his piety, Carloman could be ruthless towards real or perceived opponents. After repeated armed revolts and rebellions, Carloman in 746 convened an assembly of the Alemanni magnates at Cannstatt and then had most of the magnates, numbering in the thousands, arrested and executed for high treason in the Blood Court at Cannstatt. This eradicated virtually the entire tribal leadership of the Alemanni and ended the independence of the tribal duchy of Alemannia, which was thereafter governed by counts appointed by their Frankish overlords.
These actions strengthened Carloman's position, and that of the family as a whole, especially in terms of their rivalries with other leading barbarian families such as the Bavarian Agilolfings.
On 15 August 747, Carloman renounced his position as majordomo and withdrew to a monastic life, being tonsured in Rome by Pope Zachary. All sources from the period indicate that Carloman's renunciation of the world was volitional, although some have speculated that he went to Rome for other, unspecified reasons and was "encouraged" to remain in Rome by the pope, acting on a request from Pepin to keep Carloman in Italy.
Carloman founded a monastery on Monte Soratte and then went to Monte Cassino. All sources from the period indicate that he believed his calling was the Church. He withdrew to Monte Cassino and spent most of the remainder of his life there, presumably in meditation and prayer. His son, Drogo, demanded from Pepin the Short his father's share of the family patrimony, but was swiftly neutralised.
Seven years after Carloman's retirement and on the eve of his death, he once more stepped briefly on the public stage. In 754, Pope Stephen II had begged Pepin, now king, to come to his aid against the king of the Lombards, Aistulf. Carloman left Monte Cassino to visit his brother to ask him not to march on Italy (and possibly to drum up support for his son Drogo). Pippin was unmoved, and imprisoned Carloman in Vienne, where he died on 17 August. He was buried in Monte Cassino.