People come to your village if there is enough housing space available (build more hovels if not) and you have positive popularity. Manage your population wisely! Villagers not working at a building will still consume food and ale, so you never want to have too many idle peasants or they will quickly be eating more food than your village is able to produce.