Perhaps they are confident with their powers, they don't feel like guarding the whole place, the sinister temple is intimidating enough versus the ordinary common folk. Who knows what magical and divine natured traps lie around here, I think the denizens let the temple to overcome you instead of them.
Thinking again the champion adds.
Maybe this argument is part of the temple influence, we must concentrate...