You can gain access to more defences by upgrading the Keep within the castle. This can be done by acquiring the required amount of wooden planks and stone bricks, acquired from Woodcutters and Stone Quarries.

If an upgrade is not available, you may need to reach the required rank.