When a player advances to the next county they have to rebuild their village. As you rebuild your buildings, they will be built to the level they were in the last county.

The Manor and the castle buildings are left unaffected by the advancement to further counties.