The village name is Anglo Saxon in origin, and possibly means 'swan stream'. In the Anglo-Saxon Chronicle in 792 the village was recorded as Suanaburna.
A grant of land was made to Woburn Abbey in 1201. The first vicar of the parish arrived in 1218 and the parish church was dedicated in 1230. The abbey was dissolved in 1538 and its lands were later sold by the Crown.
Swanbourne supported Parliament in the English Civil War and was burnt by Royalist troops in 1643. The Aylesbury–Buckingham turnpike road through Swanbourne opened in 1722. Common lands were enclosed in 1762–63 and divided among 50 landowners.
Swanbourne House was bought in 1798 by Thomas Fremantle (1765–1819), for his wife Elizabeth, known as Betsey, for 900 guineas. The Fremantle family, originally from Aston Abbotts, had strong naval connections. Their eldest son Sir Thomas Francis Fremantle (1798–1890) became a prominent Tory politician. Their second son Charles (1800–1869) followed his father into the British Royal Navy and was instrumental in founding the Swan River Colony in Western Australia. This accounts for the place names Fremantle, Swanbourne and Cottesloe in the Perth area of Western Australia.
Swanbourne House is still owned by the Fremantle family trust, but let (see "Schools"). The present head of the family is Commander John Tapling Fremantle, 5th Baron Cottesloe. A former Lord Lieutenant of Buckinghamshire, he lives in the village, as does his daughter Elizabeth, the Hon. Mrs. Duncan Smith, with her husband Iain Duncan Smith, the Conservative politician.
There was an agricultural strike in Swanbourne in 1873, led by members of the Primitive Methodist Chapel who were in the National Agricultural Labourers' Union (NALU).
Attached to the village is the hamlet of Nearton End.
The first school in the village was founded in 1712 under the will of one Nicholas Godwin.
The present-day Swanbourne Church of England School in Winslow Road is a mixed, voluntary aided, primary school which operates as one component of the Three Schools, with Mursley Church of England School and Drayton Parslow Village School, under an agreement reached in 2009. At present Swanbourne has 115 pupils in four classes, aged seven to eleven, out of a total of 199 pupils at the three component schools. The Swanbourne classes are divided into sets for the core subjects. Half the pupils come from the catchment area and half from further afield.
Swanbourne House is a coeducational infants' and prep school for some 380 pupils from age three for day pupils and seven for boarders, up to the age of 13. It was founded in 1920. Anthony Chenevix-Trench taught English and History at the school for one term in the early 1970s, between being headmaster of Eton College and headmaster of Fettes College.
The private Home Farm Day Nursery is also in Winslow Road.
St Swithun's Church (Anglican), a Grade II* listed historic building, stands at the east end of the village, opposite Swanbourne House. The nave, chancel and tower date from the first half of the 13th century. The north aisle was added in the second half of the 15th century and the tower rebuilt half a century later. The church is in good repair, and contains some stained glass and a wooden ceiling, both probably dating from the 19th century. There are also some medieval carvings and the remains of three medieval murals in the north aisle. The tower contains six bells and a sanctus. There are monumental brasses on the south and north sides of the chancel. The latter, showing Thomas Adams (died 1626) and his family, bears the baneful inscription, “Who in prime of youth by bloudy theves was slain, / In Liscombe ground his bloud ye grass did staine.”
Swanbourne Baptist Church, in Mursley Road, was built in 1809, rebuilt in 1863 and closed in 1972, when it was converted into a dwelling.
Swanbourne Methodist chapel is in Nearton End. The first Chapel was built for the Primitive Methodists in 1858 and a new one superseded it in 1907. Formerly in the Stewkley Circuit, then the Leighton Buzzard Circuit, the church is now in the Vale of Aylesbury Circuit. Sunday services are at 5 pm with a Sunday School each week at 10.30 am.
There is a general store and sub-post office in Mursley Road. Also in Mursley Road is a public house and restaurant called The Betsey Wynne after the diarist ancestor of the Fremantles. It specializes in English food and drink and self-grown produce. It opened at the end of July 2006 and has won some acclaim. It was taken over from the Swanbourne Estate by Oakham Inns in 2015.
Bed and breakfast accommodation is available at the Cherry Tree in Tattams Lane.
There are limited weekday bus services between Swanbourne and Winslow, Bletchley or Milton Keynes.
Swanbourne railway station, on the Oxford to Cambridge 'Varsity Line', was open from 1851 to 1967. The site of the former station is about 1.5 miles (2.4 km) from the village and half way to Little Horwood. It was also about the same distance from Mursley. The nearest station today is Bletchley railway station, 8 miles (13 km) away.