The barony of Aberdour was acquired in 1126, by Sir Alan de Mortimer, on his marriage to Anicea, daughter of Sir John de Vipont. Sir Alan built St Fillan's Church, which still stands, next to the castle, in around 1140, and his family probably built the original hall house in around 1200, or possibly even earlier. In 1216, another Alan de Mortimer is recorded granting land to the monks of Inchcolm Abbey. There is no record of what happened to the de Mortimers, but in the early 14th century, King Robert the Bruce granted Aberdour to his kinsman, Thomas Randolph, Earl of Moray (d. 1332). Moray's grandson granted the barony in turn to Sir William Douglas of Liddesdale (c. 1300-1353), in 1342.
In 1351, Sir William Douglas gave the lands of Aberdour to his nephew, Sir James Douglas of Dalkeith, although he retained the castle for himself until his death two years later. The grant was confirmed by King David II in 1361. In 1386 Aberdour and Dalkeith were combined to form a single barony, with the principal seat at Dalkeith, near Edinburgh, and Aberdour as a secondary residence. James, fourth Lord Dalkeith, succeeded to the joint barony in 1456, and was created Earl of Morton in 1458, prior to his marriage to Joanna, the deaf-mute daughter of James I. The newly created earl expanded the existing hall house, heightening and rebuilding the structure to suit his elevated status. The second earl carried out extensions to Aberdour Castle around 1500, building a new stair tower and south block.
In 1538 James V summoned the 3rd Earl of Morton before the Privy Council, accusing him of non-payment of his feudal dues, and in 1540 he banished the Earl to Inverness. Morton reached Brechin, in Angus, where he signed a deed resigning his lands to his kinsman Robert Douglas of Lochleven. Lochleven was compelled to resign the lands in turn to James V, although he was permitted to keep Aberdour Castle. After James V's death in late 1542, George Douglas of Pittendreich and the Earl of Arran assisted Morton in reclaiming his lands, including Aberdour. In return their sons were to marry two of Morton's three daughters. Pittendreich's son James (1525-1581) married the heiress, Elizabeth, and succeeded, in 1553, as 4th Earl of Morton.
Aberdour Castle was reserved to his mother-in-law, Countess Katherine, until 1564, when Mary, Queen of Scots, confirmed Morton's right to the whole barony of Dalkeith and Aberdour. In 1566, Morton was involved in planning a rebellion against the Queen, which resulted in the murder of Queen Mary's secretary, David Riccio, but failed to gain further momentum, and Morton was forced to flee to England. However, by the end of the year he had returned, and by July the following year, Mary was imprisoned and had been forced to abdicate by the Scottish noblemen. Morton was appointed Regent of Scotland, for the child King James VI in 1572. He undertook extensions to the castle in the 1570s, rebuilding the south block of c. 1500, and extending it further south to form the present central range. He also drew inspiration from contemporary gardens in England, such as Hampton Court, in laying out the terraced gardens. The Privy Council met at Aberdour Castle in August 1576, but Morton's regency came to an end in 1578. He was later implicated in the 1567 murder of Queen Mary's husband, Lord Darnley, and executed in 1581 on the orders of the young King.
While Morton was in prison, his lands were given to his nephew, Archibald Douglas, 8th Earl of Angus, but were granted to the Earl of Lennox after Morton's execution. In 1587, Lennox returned the Morton lands to Angus, who was now acknowledged as the 5th Earl of Morton. On Angus' death, in 1588, the earldom of Morton passed to another kinsman, William Douglas of Lochleven (d. 1606). William's son predeceased his father, but his widow, Jean Lyon, continued to live at Aberdour with her third husband, Alexander Lindsay, 1st Lord Spynie (d. 1607).
William Douglas, 7th Earl of Morton (1582-1648), sometimes numbered as the 6th Earl, inherited Aberdour from his grandfather in 1606. He was Treasurer of Scotland from 1630–36, and a strong supporter of the Stewart dynasty during the Wars of the Three Kingdoms (1639-1651). However, he was compelled to spend much of his fortune in the royal interest, leading him into financial difficulty, and forcing him to sell Dalkeith to the Earl of Buccleuch in 1642. Earl William built the Renaissance east wing at Aberdour, probably around 1635. It was certainly standing in 1647, when an inventory records luxurious furniture, carpets, and tapestries. The walled garden was also built in the 1630s, and improvements made to the terraces. Aberdour was therefore a suitable principal residence for the Earls after the sale of Dalkeith.
The Earls of Morton continued to live at Aberdour, although they never regained their earlier high status. In 1688 the castle was badly damaged by fire, and in 1690 the 11th Earl consulted the architect James Smith. Smith surveyed the damage and drew up proposals for repairing and extending the castle with another wing north of the east range. The Earl also sought estimates for the demolition of the tower house and central range. Nothing ever came of these proposals, although repairs were carried out to the east range, being completed in 1703.
Government troops were stationed at Aberdour Castle during the Jacobite Rising of 1715. During their stay, a second fire again caused extensive damage. In 1725, the Mortons bought an adjacent property, Cuttlehill House, which they renamed Aberdour House, and the castle ceased to be a residence. The east range was again repaired, and was used for various purposes, including a school room, a barracks, and a masonic hall. In 1924 the castle and gardens were placed in state care, and continue to be managed by Historic Scotland as a visitor attraction. The south-east wing was initially restored as a home for the custodian, and now houses a cafe. The castle is protected as a Scheduled Ancient Monument, and the castle, dovecot and walled garden are Category A listed buildings, the highest level of protection for a historic building in Scotland. The grounds, and surrounding former parks, are listed on the Inventory of Gardens and Designed Landscapes, the national register of significant gardens.
The castle originally comprised the 12th- or 13th-century hall house, which was extended in the 15th century. In the 16th century the central range was built to the south of the tower house, and new inner and outer courtyard walls were constructed. The east part of the inner courtyard wall is reduced to foundations, but the base of a round tower and a porter's lodge survive. To the west, the inner courtyard wall remains, enclosing the former service courtyard, which comprises a brewery and bakehouse, with ovens. The east range was added in the 17th century, along the south edge of the outer courtyard. The castle was originally approached from the north, with the entrance moved to the west, along with the 17th-century gate, when the Aberdour railway line was constructed in 1890.
The tower house measures 16 metres (52 ft) by 11 metres (36 ft) on a skewed rectangular plan. The lower two storeys comprise the oldest part of the castle, which can be dated to around 1200, while the upper parts are of the 15th century. The date for the original building is based on several pieces of evidence. The cubical blocks of masonry, the flat buttress at the north-east, and the splaying, or widening, of the base of the walls, are all indicative of this date. The collapsed part of the south wall also formerly contained an early 13th-century style double-lancet window. The original structure, of a type known as a hall house, was probably of two or three storeys, comprising a first-floor hall over an unvaulted basement, and may have been surrounded by a defensive timber stockade or enclosure. The 15th-century rebuilding added two upper floors, and rearranged the interior. Vaulted basements, including a kitchen, and a spiral stair were inserted, and the heightened wall was topped by a parapet with machicolations; spaces through which objects could be dropped on attackers at the base of the wall. The tower suffered major collapses in 1844 and 1919, and today just the basement survives, with only a small section of the south-east wall standing to its full height.
The central range was begun around 1500, possibly as a two-storey building containing a great hall, although only fragments of this building remain. The south entrance to the tower house was blocked up, and replaced with a ground floor entrance from the east. A new stair tower, with a broad spiral stair, was built at the south-east corner. Originally topped by a conical roof, this stair gave access to the upper floors in the tower, and to the central range. Around 1570, the 4th Earl of Morton rebuilt the block, extending it further south to form new apartments. This block, forming the present three-storey central range, includes a vaulted kitchen and cellar in the basement, with suites of rooms above. The two bedrooms on the first floor each have their own closets and garderobes, or privies. The west apartment is accessed from the terraced garden, via another stair to the south-west, and has a private stair which led up to another chamber above, suggesting that these were the Earl and Countesses apartments. The upper storey, whose timber floor is now missing, comprised three more apartments, including one over the stair. The exterior of the block is decorated with a horizontal decorative strip of masonry known as a string course. One first-floor window is decorated with carved stone pilasters and strapwork, in a similar style to windows at Edinburgh Castle and Drochil Castle, in the Borders, both of which were built under Morton's Regency.
The east range, of c.1635, comprises a long, narrow building, with a short projecting wing to the south-east. It is the only part of the castle still roofed. The first floor of the wing is occupied by a long gallery, accessed from the central range, with stables and storerooms below. The long gallery was used for entertaining, and, in 1647, is recorded as containing 46 pictures and a harpsichord, among other furniture. In the south-east wing are three chambers, one on each floor, linked by a spiral stair. The first-floor room has a 17th-century painted ceiling, decorated with fruit, foliage, and heraldic emblems. The south-east wing has crow-step gables, and a sundial on one corner. The east window of the long gallery is decorated with pilasters and a pediment. The roof is of 18th-century date, and is lower than the original, resulting in the loss of the triangular pediments which once topped each of the side windows.
There were gardens at Aberdour Castle from at least 1540. The terraced garden dates from the time of the 4th Earl of Morton, who succeeded in 1553, and comprises four broad L-shaped terraces. At the bottom of the terraces was an orchard, laid out in 1690, and recently replanted. The extent of the terracing was only rediscovered following excavations in the 1970s, after parts of the garden had been in use as a market garden. Archaeological investigations were undertaken between 1977 and 1980, to determine whether the lower terraces, which were depicted on a map of 1740, had indeed existed. Although the foundations of the terraces were found, the remains could not be precisely dated, but are thought to have been constructed in the second half of the 16th century. The high retaining walls were rebuilt in 1981, and the terraces were laid with grass, since the excavations had not revealed any evidence of historic planting schemes.
A 16th-century "beehive" shape doocot, or pigeon house, is located to the south, containing around 600 nesting boxes. The 9 metres (30 ft) structure rises in four steps, divided by "rat courses"; projecting ribs which prevented rats from climbing inside.
The 17th-century walled garden covers around 5,000 square metres (1.2 acres), with walls up to 4 metres (13 ft) high. It lies to the west of the outer courtyard, and was originally entered from doors in the south-west and north-east corners. These doors have carved pediments, with strapwork and the Douglas heart emblem over the west door, which led to the terraces. The east door led to St. Fillan's Church, and is carved with the date 1632, together with a monogram of the initials of the Earl and Countess. A kitchen garden was located across the Kirk Lane, and bee boles are located in the walls. A summer house was built into the south-east garden wall in 1675, but was demolished in the 18th century. In 1691, botanist James Sutherland supplied exotic plants, including Persian jasmine, tamarisks and figs, to Aberdour from the Physic Garden in Edinburgh, the forerunner of the Royal Botanic Garden. The later entrance in the west wall dates from around 1740. During the Second World War, the garden was in use as a market garden and for pig rearing. In the centre of the garden, an early 17th-century polygonal sundial is mounted on a 19th-century base. The sundial was brought here in the 1970s, possibly from Castle Wigg in south-west Scotland.
Originally named Cuttlehill, the house was built in the 17th century, and is dated 1672. It was enlarged in 1715 by its then owner, the Earl of Moray. The Earl of Morton purchased it in 1725, finally moving out of the dilapidated castle. In 1731 the 12th Earl consulted James Gibbs on improvements. It is uncertain how much of Gibbs' proposals were executed, although the doorway has a surround in Gibbs' style. By the 20th century the house was disused, and was redeveloped as private flats in the 1990s. The house is Category A listed, and formerly had its own extensive gardens, which included the surviving 12-metre (39 ft) high obelisk of 1744–45, built by the 13th Earl to be visible from his estate at Dalmahoy, across the Firth of Forth.