According to tradition, the site has been occupied by castles since the time of Pippin of Herstal (7th century). In the later Middle Ages, the area fell under control of the Beaufort family, which oversaw the construction of a stronghold here in about 1230.

The present castle, in the form of an irregular pentagon and flanked by six round towers of varying size, dates largely from around 1410. Successive restorations modified especially the walls of the inner courtyard, one of which is lined with a distinctive half-timbered gallery of two levels, and another of which was given a red brick facade in the Louis XV style. The northern frontage is crowned with a small cupola containing a clock.