iserith
Magic Wordsmith
It's an interesting experiment but I think the way the goals are set might be the elephant in the room.
You will presumably be defining a metric that lets you place a break point between "DMG guideliness are okay" and "DMG guidelines are not okay" in terms of resource depletion (in the most general sense) or some other parameter, but the DM has the power to move the result above or below the breakpoint by tweaking the encounter goals whilst still complying with the guidelines.
For example, suppose the DM places an encounter consisting of an Archmage allied with a Beholder. According to the guidelines, that's a Hard encounter. But there's a world of difference between monster goals of
and so on, in terms not only of how the encounter will play out but in terms of what can be judged successful or unsuccessful in terms of party resource depletion.
- Put all the PCs to sleep so the Archmage can steal all their magic items and run away
- Delay the party by 4 rounds even if you both die in the attempt
- Kill the dwarf at all costs
Remember though that the Creating Encounters guidelines (starting on DMG page 81) includes character objectives which aren't limited to Kill All Monsters. A previous section on building adventures talks about villains and their particular motives and actions they take to meet their goals. These need to be taken into consideration in my view when building engaging challenges within the constraints of the recommend XP budgets and CR.