Looking at the map again, the main temple is on the way to the kitchens. We will have to be quiet going past them. I doubt we'll want to tangle with the high priestess first. So my original plan still stands, we just have to be aware of sneaking past the temple.
The other issue with the plan is what do we do with the prisoners who won't come with us. Trust them to stay quiet and undiscovered?