This is probably caused by your popularity being low. Ensure you have placed a granary (press G to locate it) and that you have food being produced. Also make sure you are not taxing your people too heavily as this can also cause people to abandon you.