The Dovecot Tower was a decorative addition to the Castle, along with the Gatehouse and Outer Wall, built by the Mortimer family c.1380.

Edmund had married Phillipa, grand-daughter of Edward III. In the gatehouse was born Usk's most famous son, the Welsh chronicler Adam of Usk whose memorial is in St Mary's parish church. This was the church of Usk Priory, which housed an Abbess and nuns of noble birth. The town with its medieval streets, and later inns, shops, museum and prison was built on the site of the Roman fortress of Burrium AD 55.