This is probably caused by your popularity being low. Ensure you have placed a granary (press G to locate it) and that you have food being produced. Also make sure you are not taxing your people too heavily as this can also cause people to abandon you.
Why are people leaving my castle ?
Last Updated: 1456d