Doesn't this become something of a "quantum ogre" problem? If you have a cave from which raiders emerge to harass passing caravans on the King's Road, but only decide whether they are goblins, bugbears or ogres depending on when the PCs bother to confront them, how do you make them inherently...