Let's not let facts intrude on the "it's the players' fault for playing the game by the rules and realizing their own powers are always strictly better than the max benefit of improvisation" -argument.
It was absolutely a systemic rules problem. Improvising should have a chance of killing an enemy outright, otherwise your normal abilities will probably kill the monster anyway within a round or two. As a guy who loves D&D improv and stunts, it was mathematically undeniably inferior to not use your own powers, which were often augmented by feats, class features, magic items, and synergy with class features, feats, and magic items owned by other members of your party.
If the enemy had received cold vuln last round, are we expected that anyone in their right mind wouldn't take advantage of that? Part of being a team player is working with the team. Bypassing the normal rules for powers often just means you are throwing away 100% of the synergy you could have had.
This is bad game design, IMO. The problem was an overly complex inter-locking set of benefits that build upon one another, like when you do a basic attack you gain this benefit with this weapon after you took this feat and class feature, so that the warlord grants you a basic attack it will trigger all those things. If the warlord doesn't do that, he's throwing all that possible optimisation away. The quote from the 4 DMG doesn't even come close to realizing all the lost synergy. It was poorly thought out. Players aren't stupid, they were playing the game well, from not only a meta-game optimisation standpoint, but also an in-game party synergy "team player" standpoint, by not using P42.
Picking an inferior mechanical choice repeatedly (even ever) doesn't make you a better roleplayer, it just means the system is badly designed, objectively speaking.
Doesn't this argument essentially apply to all editions of D&D, though? That you should, logically, optimally, only improvise if it's better than using your powers?
I mean, this is why Thieves and Fighters tended to improvise most in 2E AD&D, because they didn't have any powers, and it was often possible to come up with an improvisation better than an unreliable (due to low THAC0) backstab or even the Fighter's multiple attacks. Whereas a Wizard of about level 7-9+ in 2E (with a high INT) rarely improvised (as in tried something other than casting a spell), because spells were almost always the best option.
5E will definitely have this problem, too, if you consider it a problem - unless the DM is willing to make improvised actions into instant takedowns, as you suggest, it's rarely going to be smart to use them.
Using P42 would get you a lot of dirty looks at the Encounters tables I took part in, because the party eventually realized you weren't pulling your weight compared to what you could be doing using your AEDU routine.
Are you going to put the entire forum on ignore? You are wrong.
What am I wrong about, specifically? Are you saying that Page 42 applies specifically to Underwater and Mounted Combat?
I'm sorry that people an Encounters behaved nastily to you if you didn't play in a hyper-optimized way, but that's outside my control. I have not suggested that Page 42 is optimal. I don't play with a group who only do "optimal" things, and if a player was nasty to another just because the other wasn't "using his AEDU routine", dude, I'd kick him out of my group.
EDIT - I should add that any DM worth their salt should recall that Page 42 is "guidelines", not rules - you don't actually have to use those numbers. If you feel they're too low for the optimization-level of the group, you can increase them. If they're too high, well, decrease them. I think your complaint is Encounters-specific, in a sense, because there the DMs can't really change anything.
EDIT EDIT - Also, yes, picking a mechanically worse option totally can make you a better roleplayer. A very simple example is if you are playing a character sworn to not use guns in a system where guns are mechanically superior to other weapons - you could, of course, ignore the fact that your character doesn't use guns, and just use them, because they're mechanically superior, but would be terrible RP. Apparently in your book, it would "make you a better roleplayer", by ignoring RP and instead going for mechanical optimization. That's... an unusual viewpoint, no?