Building up a strong castle is always recommended, but with pillage and ransack attacks you need to build a bit further out as once attackers get inside the white line, each attacker inside continually reduces the success counter by one. Building further out will help you target incoming troops with your archers more quickly.