Even these meta-game level designs of treasure control and giving a "screw you" to people listening at doors, the monster stats themselves were still based relatively on their fictional description. Rust monsters have the HP and AC they do because they have tough hides (but not too tough) rather than because the PCs happen to be level 7 or something when they finally fight one.
I'm not really satisfied with this approach as long as the stats of a monster are a function of it's level and role/type rather than it's fictional description. At a given level, everything about a monster can be only one of a handful of different stat blocks, then differentiated only by its powers.
I agree with you about the relationship between a rust monster, its hide and its AC in classic D&D compared to 4e. What this means in AD&D and B/X is that at low levels there is a fairly low hit rate by the PCs (many things have ACs in a range of 4 to 7, and that's somewhat hard for a low level PC to hit - 13+ on a d20 before mods), while at high levels there is a pretty high hit rate.
This is obviously very different from 4e, which is designed around a more-or-less constant hit rate, and has a whole condition-infliction sub-game going on which only works and makes sense with that more-or-less constant hit rate.
On this design issue, my personally least favourite edition is 3E, which invented the "natural armour bonus" to get scaling defences something akin to 4e (although I gather not quite as smooth), while pretending to have the tough-hide simulation of classic D&D. (I mean, when natural armour is tougher than a mithral breastplate, what the heck am I meant to be envisioning in the fiction?)
It doesn't matter if a goblin is naked or wearing full plate armour. It's AC is determined by it's level and role, fiction be damned.
But this I don't agree with - taking off its armour should cost it a level or two. (It's attack bonus drops as it has to fight more defensively to make up for it's lack of armour, and its hp drop because it's less resilient without full plate.)
Or, if you don't want to bother with the maths of this, then if its full plate is lost you have to narrate something else in to plug the gap (of course this raises issues about railroading, but that's another issue).
The 4e DMG says a bit about this (pp 174-75):
You can add equipment to a monster to make it a little more challenging . . .
Remember that a monster’s game statistics are set to be appropriate for its level. Thus, altering a monster’s attack, defense, or damage values is a lot like changing its level (see above). Avoid the temptation simply to give all your monsters better armor and weapons. . .
If you want to give a monster equipment that changes its attack, defense, or damage values by more than a point or so, consider also making those alterations as part of changing its level.
More advice, and examples of this sort of thing in published adventures, would help.
4E has the illusion of an actual leveling system, but the % of HP you do with an average attack, the to-hit number needed, the die roll needed to meet a DC, none of it actually changes. It's all a farce.
Like I said above, I think this is a little harsh. The
fiction changes. And in a game of shared exploration of an imaginary setting, that's far from irrelevant. (4e is something like a more byzantine version of HeroQuest revised's pass/fail cycle.)
The rulebooks - both PHB and DMG - even talk about this, in their discussion of the different tiers. Again, though, I think more could have been done to bring out the ways in which these changes in the fiction matter. And more could have been done to explain how paragon paths and epic destinies, which are key points of expressing these fictional developments on the character sheet, feed into skill challenge framing and resolution (eg in persuading the duke, surely it makes a difference whether you're a Questing Knight, a Battlefield Archer or a Demonskin Adept - just to point to 3 of the PCs in my own game).
My own impression - and it's really nothing more than a gut feel based on reading these boards for a few years - is that a lot of people play 4e
without treating the fiction as anything but colour. WotC's 4e modules tend to give this vibe. When the fiction becomes mere colour, and levelling is not a reward (for the reasons you give), and so action resolution becomes an end in itself, then accusations of "tactical skirmish game", "boardgame" and of being farcical do have some force, in my view.
It can create problems when you don't have tactical combat as a discrete game mode that you enter and exit. Replaying through Keep on the Borderlands with BECMI has definitely been an eye opener. Between wandering monsters and staying the exploration game mode for much of the build up to combat means managing HP across multiple encounters becomes the focus rather than within the individual tactical combats.
Before I switched to playing BECMI, I had done some serious rules hacking of 4E to get it to produce the type of play I was looking for. Given the difficulty of excising the pacing mechanic and revamping the refresh rates based on encounter or daily power usage, healing surges, etc., I'm not sure D&D Next can support both a OD&D/BECMI/1E style approach to resources and a 4E one, even with modules added or removed.
The other issue I have is the tactical encounter rather than a continual exploration-description mode of play.
At least in my view, 4e is pretty clearly designed for a "scene framing" approach, rather than a continuous exploration approach, to play. And on that I think we're in agreement. (Although I think it doesn't have to be tactical encounter. There are also skill challenges. And even semi-free-form exploration can be done in a scene-frame-y sort of way, in my experience. But these minor quibbles don't detract from the broader point.)
I think we also agree that the features of 4e that make it support its approach - like the constant hit rate in combat, the condition-infliction subgame, encounter powers, healing surges, no long duration effects, etc - are all at odds with continuous exploration, precisely because they confine the significance of events to the scenes in which they are framed, or their implications for newly-framed scenes.
Can D&Dnext somehow support both? I agree it's tricky. But look at Burning Wheel. Take out Let it Ride, take out the Intent and Task guidelines, and you've got something that looks a bit like a dice pool variant of Runequest. That might still be a pretty playable game.
Now I think that 4e and classic D&D are even more different than BW and this imaginary BW variant - but I still think this shows that the task isn't necessarily hopeless, provided the designers can find just the right points in the design where a subtle nudge, the doesn't change any of the raw numbers too much, can make a big difference in the way those numbers feed into action resolution and its consequences.