James Butler, born on 24 November 1420, was the eldest son of James Butler, 4th Earl of Ormond, by his first wife, Joan de Beauchamp, Countess of Ormond (d. 3 or 5 August 1430). He had two younger brothers, John Butler, 6th Earl of Ormond, and Thomas Butler, 7th Earl of Ormond, as well as two sisters, Elizabeth Butler, who married John Talbot, 2nd Earl of Shrewsbury, and Anne Butler (d. 4 January 1435), who was contracted to marry Thomas FitzGerald, 7th Earl of Desmond, although the marriage appears not to have taken place.
He was created Earl of Wiltshire, in the Peerage of England, by King Henry VI on 8 July 1449, for his fidelity to the Lancastrian interest. In 1451 he became Lord Deputy of Ireland; the next year he succeeded his father in the Earldom of Ormond and was additionally appointed Lord Lieutenant for ten years. In 1454, he was constituted Lord High Treasurer of England and made a Knight of the Garter in 1459.
By virtue of the rights of his wife, Avice, in the manor of Frome he exercised patronage of the chantry of St Andrew in the parish church at Frome in appointments made in 1452, 1453 and 1458.
When the dynastic civil wars - known as the Wars of the Roses - broke out, Wiltshire fought on the Lancastrian side, becoming one of Queen consort Margaret of Anjou's staunchest supporters. He was present at the First Battle of St Albans in 1455, Battle of Wakefield in 1460, Battle of Mortimer's Cross in 1461 and at the Battle of Towton. He died on 1 May 1461, beheaded at Newcastle by the Yorkists after the Lancastrian army was soundly defeated at the Battle of Towton.
He was succeeded by his brother, John Butler, 6th Earl of Ormond.
He married firstly Avice Stafford (1423-1457), daughter and heiress of Sir Richard Stafford by Maud Lovell, daughter and heiress of Robert Lovell, esquire, by whom he had no children.
He married secondly Eleanor Beaufort (1431-1501), daughter of Edmund Beaufort, 2nd Duke of Somerset by Eleanor Beauchamp, Duchess of Somerset, daughter of Richard Beauchamp, 13th Earl of Warwick, by whom he had no children.
The Earl of Wiltshire was described as the most handsome man in the Kingdom and as Gregory records, at the First Battle of St Albans in 1455, Wiltshire "fought mainly with the heels, for he was frightened of losing his beauty"; prudently he had taken off his armour and hidden it in a ditch, donning a monk's habit.