Nail said:
The main reason, IMO, is "equipment", rather than powers like regeneration. A
significant fraction of a PCs durability (as distinct from "power"), comes from the PCs wealth/equipment. Add that to a monster's abilities, an' ya gots trouble.
That's another good point. Creatures in the MM have treasure (I'm not a DM, so I never figured out how much standard tended to be) but this is more like "you kill the monster and find an unused magic ring/cool painting when you search it." For classed NPCs (or monsters with class levels) they have stat-boosting items, magic weapons and armor and actually use their treasure. I remember one time many years ago when the DM rolled up treasure after we killed an enemy and rolled a strong item that would have made it immune to the attack that killed it.
Looking at it now, a CR 14 dragon will have 50k of treasure. If this was all in magic items, it would have something like (slots permitting), +2 Str, +2 Dex, +2 Con, +3 Armor, +1 Natural Armor, +1 Deflection, +2 Resistance, +1 Amulet of Mighty Fists, Boots of Speed and a few thousand gp of potions or scrolls. That is +7 AC, +3 to hit, +1-2 to damage, +2-4 to saves, +20 or so HP, and an extra attack a round.
The marginal value of the first 50k of gold in magic items is incredible, as you can see here. AC bonuses are particularly easy to get, Boots of Speed allow more attacks, and so on. The durability part you mentioned is here as well. For another 50k, a +1 Ioun Stone of Insight, upgrading the Resistance to +3, upgrading the Natural Armor to +2, upgrading the Deflection to +2, and upgrading the Strength to +4 are all very valuable. I’m still under 85k and this dragon already has +10 AC from using its treasure as magic items. Add to this the fact that AC increases help you more the higher your AC is, and this villain will be exceedingly difficult to kill.
Eventually, the quadratic formula for item pricing makes increased treasure less worthwhile, but monster/NPC treasure is low enough that diminishing returns isn’t reached quickly.
NPCs also get less and less treasure compared to the PCs as the levels increase. While quadratic pricing helps makes this not as insurmountable of a power decrease, it becomes a large factor for a character who relies on numbers (to hit, AC, saves). If a PC gets the treasure for 2 NPCs of his level each time he advances, you can see where the problem lies. If NPCs had half the treasure the PCs did, you would have to double your treasure each level. Over time, for PC treasure to stay non-exponential, NPC treasure has to become a smaller and smaller fraction of the PC’s treasure.
The ELH noticed this and even commented that the CRs for level 21+ NPCs should be adjusted down as a result. If I attempted to solve this problem, I would probably give more innate types of abilities to non-monster classed NPCs (so, +2 strength by evil blessing instead of Gauntlets of Ogre Power). This would let me avoid outfitting every NPC with the magic items listed above. This way character wealth could stay manageable and non-monster NPCs could be more competitive.