This means that your population should be rising but is not, due to a couple of factors. These factors could be:

a) You have reached your housing limit and you need to place more houses before your population will continue to rise.

b) You have the housing space available but you have filled up the available space around your campfire and you will have to give these people jobs before more people will arrive at the castle.