Clicking on the popularity icon will open the village management controls. Here you can adjust the tax rate (lower is generally more popular), set rations and adjust ale consumption rates if you have an inn and some ale. These will have a direct effect on popularity. How crowded your housing is also has a bearing on your popularity, placing more houses is usually a popular move. Other factors are both popular & unpopular buildings that you place and external events such as the weather, or enemies in your parish for example.