After the castle was left to decay over several centuries, the ruin was used as a quarry for the construction of the town hall of Skanör at the end of the 18th century. For this reason, the remains of the castle today consists essentially only of the manmade castle hill and a moat.
Skanör Castle consisted originally of a two storey brick building of 16x8 meters and several smaller buildings. The castle was surrounded by two moats, where the outer moat was additional surrounded by a palisade wall.
The origin of the castle is closely connected to the medieval Skånemarknaden (Skåne market), a large fish market which was held annually in the towns of Skanör and Falsterbo between the 12th and 16th century. This market was the most important source of income for the Danish crown at its time. Therefore, the substantially tasks for the royal governors were to ensure law and order on the market and to collect taxes and fees from market participants.
1311 the castle was conquered and destroyed by armed forces of the Wendish Hansa, but was quickly rebuilt. A few years later it was destroyed once more, but this time by a Swedish army (Skanör belonged to Denmark at that time) and then was rebuilt again.
With the relocation of the Skånemarknaden towards Falsterbo, the much better fortified castle Falsterbohus became the seat of the royal bailiff at the end of the 14th century. The castle in Skanör was abandoned and left to decay.