Not at all really. A dungeon could feature humanoids, giants, demons, magical beasts, and aberrations quite easily. The giants and their humanoid slaves are digging a mine to recover something for their demonic masters aided by destrarchan and delvers. Now, any given encounter could feature one or all of the above. That means you may very well be swapping in this or that gem in the middle of combat. Particularly if a dungeon is reactive and reinforcements start coming in.
Add in a couple of constructs and some other goodies floating around and you've got a fair number of different types all working together.