Well we're stuck between a rock and a hard place with regards to in-game playing versus metagame knowledge. On the one hand... from an in-game perspective, the group should basically just leave because this is someone's home and they told us to get out. For our paladins especially, trying to justify why we'd wander through the castle when we've been told not to is a big dodgy.
But at the same time... the game doesn't continue until we find the tome, plus do everything that needs to occur at the chapel. So while going back to town is the "normal" and "right" thing to do... it basically stops the game's progress.
We either need to metagame and say "well, we need the book and the chapel so let's just blow off what Sergei said and start wandering through the castle to do what we need done"... or we go back to the village and try and concoct some reason to justify invading the castle. Which seems a bit of a waste of time, but would be the only thing that would make in-game sense why we'd tresspass on someone's property.
While us players know the endgame of the campaign is back at the castle... our characters really don't. So all they have is that the castle seems evil, and they have no way of knowing or finding out if the owner is who he says he is. The only reason I put forth the idea of Jarrith trying to sneak through the castle to the library is because that character can justify doing it, plus by sneaking through he has less of a chance of setting off a giant war inside the castle if it's infested with all kinds of nasties.