The site of Cuzorn, located in a bottleneck of the Lémance valley, allowed control of access to the region, but its origin remains unknown. The first mention of Cuzorn dates back to 1242, in the ruins of the Belleperche abbey, where an inscription mentions a certain B. de Cuzorn.
In 1259, the local nobility paid homage to the Count of Toulouse, including the brothers Gaubert, Pons, Guillaume, and Gaillard de Fumel, although neither lordship nor castle were mentioned at Cuzorn. However, documents from 1270 mention the lords of the castle of Cuzorn and Amalvinus de Cuzorn in conflict with the Bishop of Agen, suggesting a construction of the castle between 1260 and 1270.
Over the centuries, the lordship of Cuzorn had several owners, including the de Lézergues family and later the Gontaut de Saint-Geniès. In 1442, the castle was besieged, and later its defenses were dismantled. Subsequently, the lordship passed between different families until the 19th century.
In 1953, the castle was acquired by Georges Rastel, who undertook significant renovation work. In 1995, Thierry Delrieu became the owner of the castle, while the ruins were transferred to the municipality in 1996. In 1950, the castle was listed as a historical monument.
Inverness Castle sits on a cliff overlooking the River Ness. The red sandstone structure evident today was built in 1836 by architect William Burn. It is built on the site of an 11th-century (c. 1057) defensive structure. Today, it houses Inverness Sheriff Court.
The castle is said to have been built by Máel Coluim III of Scotland, after he had razed to the ground the castle in which Macbeth of Scotland according to much later tradition, murdered Máel Coluim"s father Donnchad I of Scotland, and which stood on a hill around 1 km to the north-east.
The first Inverness Castle was partially destroyed by King Robert I of Scotland and a replacement castle was sacked in the 15th century by the Clan Donald during the Siege of Inverness (1429). The castle was occupied during the Raid on Ross in 1491.
In 1548 another castle with tower was completed by George Gordon, 4th Earl of Huntly (1514–1562). He was constable of the castle until 1562.