The first document explicitly mentioning the Castle of Cancellara dates back to the 15th century. During that period, the feudal property belonged to King Ferdinand I of Naples, also known as Ferrante, and his son Frederick I of Naples. Before this date, some documents testify to the existence of the fiefdom and the settlement of Cancellara, although they do not explicitly refer to the castle.
The devastating earthquake of 1694 caused significant damage and left it in ruins. In 1806, it was partially used as a prison. In the following decades, the fortress, upon which it was built, suffered landslides, floods, and earthquakes.
The castle, constructed from limestone, spans three levels and consists of various buildings around two inner courtyards. The castle still retains architectural elements typical of medieval fortifications that evolved in response to social changes and housing needs. The original keep, likely built by the Normans, saw the addition of the castle/residence of the feudal lord during the 15th century. It was later transformed between the 16th and 18th centuries, first into a baronial palace, then divided, and finally repurposed as a prison, school, and private residence in the 19th and 20th centuries.
Celje Castle was once the largest fortification on Slovenian territory. The first fortified building on the site (a Romanesque palace) was built in the first half of the 13th century by the Counts of Heunburg from Carinthia on the stony outcrop on the western side of the ridge where the castle stands. It had five sides, or four plus the southern side, which was a natural defence. The first written records of the castle date back to between 1125 and 1137; it was probably built by Count Gunter. In the western section of the castle, there was a building with several floors. Remains of the walls of this palatium have survived. In the eastern section, there was an enclosed courtyard with large water reservoirs. The eastern wall, which protects the castle from its most exposed side, was around three metres thicker than the rest of the curtain wall. The wall was topped with a parapet and protected walkway.