If they are keeping watch that means they have to leave the safety of the hut. Since 2-3 casters are creating huts, they can't leave. Hence only a portion of the party can defend.Many of these seem to assume the party aren't keeping watch while in the hut; remember that in all versions those inside can see out and thus might be alerted to some of this stuff.
That said, if the party don't bother keeping watch...![]()
But it also depends on how sneaky the monsters are and the environment. Fail a difficult perception check to notice the spy that is checking up on the missing guards? Well they don't engage right away, they just run away to get reinforcements.
For that matter, the monsters can just pull the old trick of making so much noise just out of sight that it's difficult to sleep.

There are a lot of ways of dealing with this.