Licence granted in 1259 to Richard de Clare to build a castle at Suthwaude. A 'castle' of some form may have been built but is likely, given the lack of later evidence, to have been small and relatively domestic. The manor had just been obtained from Simon, Abbot of St. Edmunds in exchange for Mildenhall. The licence may have functioned as a legal and royal confirmation of this exchange. Site not known but possibly on site of C18 manor house at TM506762 or lost to coastal erosion. The manor and, presumably the manor house, became vested in the king in 1495 and may well have been given to the borough when this was incorporated in 1504. There was a major fire which destroyed considerable parts of the town, including the town hall and prison, in 1659. Whatever remained of the medieval manor house including any work by Richard de Clare would then have been thoroughly robbed to help with town reconstruction.
Given the origin as a monastic holding it seems likely the manorial centre was close to the church at given map reference.