The castle was built by Murdoch, Duke of Albany, who was Regent of Scotland from 1419 until 1424, during the captivity of James I in England, and who was executed by James on his return to Scotland in the latter year. The castle then became crown property until James IV bestowed it on his queen, Margaret Tudor, sister of King Henry VIII of England. It once again passed into Stewart hands and has since remained the property of the Earls of Moray, one of who's subsidiary titles is Lord Doune.

The castle, which is now a ruin, is constructed as buildings round two sides of a courtyard, dominating a strong position on a tongue of land at the confluence of the rivers Teith and Ardoch. The entrance to the quadrangle is through an archway tunnel beneath the great hall, a defensive portcullis being operated from the large window recess immediately above it. The hall is 44 feet long by 26 feet wide, and is 24 feet high to the top of the semicircular vault.

During the '45 the castle was in Hanoverian hands and was used as a strong prison for troublesome Jacobites. Based on the historical records on this point, Sir Walter Scott features the castle in Waverley, when he makes his hero spend a night in one of its splendidly damp and unpleasant dungeons. It is open to the public.