The present Beaufort castle is a Baronial style mansion built in 1880, but incorporates older building work. There has been a castle on the site since the 12th century. Beaufort is the traditional seat of the Lords Lovat. The earliest mention of the site, as Downie or Dounie Castle, occurs in the reign of Alexander I (1106–1124), when a siege took place. The original castle was built by the Byset family. The castle came into the hands of the Frasers in the late 13th century. English forces besieged the castle in 1303.
In the 1650s Dounie was attacked and burned by the forces of Oliver Cromwell during their invasion of Scotland. It was again razed by the Duke of Cumberland in Jacobite Rising of 1745, and the estate was declared forfeit. In 1815 the estate was inherited by Thomas Fraser of Strichen (1802–1875), who was reinstated to the Lordship of Lovat in 1854. In 1839 he commissioned William Burn to extend the house, and also improved the grounds and estate. His son Simon Fraser, 13th Lord Lovat (1828–1887), built the present Beaufort Castle, to designs by James Maitland Wardrop, incorporating part of the 18th-century house.
The Baronial mansion incorporates a private Roman Catholic chapel. The remains of Dounie Castle stand beside the house, and comprise a single wall, 11 metres long and 1.5 metres high.
References:The Broch of Clickimin is a large and well preserved, though somewhat restored broch near Lerwick. Originally built on an island in Clickimin Loch (now increased in size by silting and drainage), it was approached by a stone causeway. The water-level in the loch was reduced in 1874, leaving the broch high and dry. The broch is situated within a walled enclosure and, unusually for brochs, features a large 'blockhouse' between the opening in the enclosure and the broch itself. Another unusual feature is a stone slab featuring sculptured footprints, located in the causeway which approached the site. Situated across the loch is the Clickimin Leisure Centre.