You cannot expect CR to know what your players are capable of. Player skill makes an
enormous difference in encounter difficulty, much more so than DM skill. Combat-wise, most monsters are sacks of hit points with an attack roll; PCs are far more complex. Furthermore, a typical encounter has maybe 2 or 3 different monster statblocks involved (there are often more than 3
monsters, but seldom more than 2-3 statblocks), whereas each PC brings a different set of tools to the table.
All this means that the players have both more room to screw up and more room to excel than the DM does*. Just saying "equal player and DM skill" isn't enough. You have to consider how high that skill level is. The higher the overall skill level, the tougher the encounters will need to be. That isn't a flaw in the CR system, it's just inevitable. Everyone agrees that 4E set the standard for reliable, useful encounter guidelines, but even in 4E it was necessary to adjust for player skill.
The requirement for a CR system is not
accuracy but
consistency. 4E's encounter guidelines were not accurate; encounters that 4E told me were appropriate and challenging (equal level) were in fact easy and boring for my group. But 4E's encounter guidelines were
consistent. I quickly learned that, for my group, equal level meant easy and boring; level+1 meant a fun but not threatening fight; level+2 was a solid challenge; and level+3 to level+4 would really make 'em sweat**. That's what I needed. Once I had a few encounters under my belt and could assess the "player skill adjustment" for my set of players, I could plan encounters and know what I was getting into.
Consistency requires the following:
- Two monsters of the same CR present the same threat. Two encounters of the same calculated difficulty present the same actual difficulty (even if calculated does not match actual).
- For a given group of players, it should be possible to determine a "player skill adjustment." If you determine that your group, fully rested, can handle a "Deadly" encounter with a small loss of resources, you should be able to extrapolate that "Deadly times X" will be tough but manageable and "Deadly times Y" has a significant risk of PC death.
- Player skill adjustment is independent of PC level. If a "Deadly" encounter means a small loss of resources when the players are playing 4th-level characters in the Tuesday game, then a "Deadly" encounter should mean a small loss of resources when they're playing 17th-level characters in the Saturday game.
- Monsters in the Monster Manual are on par with the custom monster building guidelines in the DMG.
So, the question for 5E: Are CRs consistent? That's a real question, I have yet to run a long 5E campaign--just a few short adventures with different people. It sure doesn't look like the DMG monster building system is consistent with the MM, but do the other points hold, at least?
[SIZE=-2]*Tactically speaking, at least.
**Although even in 4E, the guidelines got less and less consistent as the party reached higher levels.[/SIZE]
