So, essentially you're saying that 3e succeeded because nearly every 3e fan was so enamoured to ... what... the flavour... that they chose to rip apart the entire baseline of the system in order to play a game that you're claiming doesn't work out of the box.
No. 3e works mainly because of skills, feats, the standardization of class advancement and multiclassing, allowing previously unseen flexibility and transpanrency in character creation. From a DM's perspective, it works because those same things apply to monsters (moreso in 3.5). Those are the baseline of the system. That flexibility gives you the most playstyle-neutral mechanics D&D has, and thus the toolbox edition. None of those have anything to do with CR, or the assumption of a four-character 25 point buy party where every character is the same level and they face balanced encounters for their level. The best thing about that assumption is indeed that it's easily ignored.
I think the success of 3e has a lot more to do with the fact that 3e works, out of the box, that balanced systems are a thousand times easier to play than unbalanced ones, and that the average gamer is savvy enough to know that a game that works is better than one that needs to be entirely rewritten.
I don't think the out of the box aspect is a huge deal for D&D. For another game it might be, but most people will play D&D even if the immediate reaction is not good just because it's D&D; they'll either adapt to its problems or fix them. D&D has a long leash.
This sort of play is of little interest to me. I may be missing something, but it suggests a very strong degree of GM force over the plot: the PCs encounter something which can only be resolved by deploying the GM's placed "macguffin". Maybe there was some other feature of the encounter to support player protagonism that you didn't mention in your write up.
If you think my players didn't take initiative you didn't read the example very well. If I threw in everything their characters had done up to that point, you'd probably see more, but that's kind of impractical. In any case, it's just one example. And I've yet to see the "flexible monster creation destroyed my game" example or even the "encounter based design is totally fun" example.
However, given that the DM does control all characters and events that are outside of the PCs' direct control, there is always going to be a "very strong degree of GM force", isn't there? In any case, I'll take "GM force" over "game designer force" any day.
It's not as if your preferred approach to monster buidling, as set out in this thread, is playstyle neutral. It seems oriented towards an approach to play in which I have very little interest.
It's as neutral as it gets. You spend as much or as little time building the monster as you want. You can plan with maximum depth or you can improvise with unprecedented ease. You can build monsters as flexibly as PCs, meaning you can make them do whatever you want. What playstyle is not supported by that?
My experience is the opposite: if the players (via a successful Monster Knowledge check for a PC) learn the abilities of a creature, and (from this) can infer to its level, and hence the mechancially measurable degree of challenge that it poses, this can certainly make them scared! Depending on what they learn, of course.
I don't "scare" my players by saying scary things to them, or telling them that what they see is scary. I scare them - and generate related emotions, like excitement - by putting their PCs in situations which will require clever and challenging play to resolve.
People are afraid of the unknown.
I don't scare my PCs by "putting their PCs in situations which will require clever and challenging play to resolve" of by telling them that what they see is of a certain level or manner of challenge; I scare them by putting them in situations that none of us know how they will resolve or whether or not they will resolve. In D&D (as opposed to in some rpgs), characters are so powerful that they usually come up with something, but the possibility of different outcomes, including failure, is still present. Not knowing what is going on or what will happen is actually essential. D&D isn't a puzzle to be solved.
If I knew that a certain battle was expected to be defeated by my party in six rounds, I would roll no dice, tell the party they beat the enemies in a short battle, and move on to the good stuff. Such a battle would certainly not empower the PCs, nor the DM; it reflects designer fiat. Whoever wrote the monster has apparently already decided what will happen at your table. Not coool.
Not that scaring them is the only style of play; I do that every now and then (but a "boss fight" is usually going to head in that direction).
I'm not sure what you mean by "balanced". If you mean that encounters posing a mechanically measurable level of challenge are neither adventurous nor interesting,
Pretty much.
Those kinds of measures are either an illusion (in that they don't take into account the complexity of the scenario and don't truly reflect its level of challenge) or a limitation (in that they impose a style of play on a freeform game with diverse players-bad idea). The CR system and its associated assumptions are an illusion, and I think at this point, most of us have made our saves to disbelieve it.
And this is me asserting my belief that I and most other DMs I know would literally rather gnaw off our own right arms than work under the system you prefer with having to custom-craft almost all monsters based on a half-finished monster-manual. We have jobs, we have lives, and we have other hobbies - and some of the burned-out 3.X DMs I know spent at least a dozen hours preparing per session at mid level and have described it as a full time job.
Well you're free to do so.
That said, having the tools laid out in front of you makes it so much easier to skip out on prep work and improvise. It's easy to take monsters and hack them on the fly when they have predictable rules for advancement. There are tons of published adventures and sources of premade stat blocks other than monster manuals for people who want to go that route. Some people may have been burned out DMs, but no one, especially not Skip Williams, ever forced them to be so.
Personally, I looked at the 4e monster manual, and the thought of having to break everything apart and rewrite the entire thing just to make it marginally usable was not appealing. That would have been a full-time job. Thankfully, no one made me DM it.
So while most DMs you know might have that opinion, I'm not particularly persuaded by them that the entire system that in no way lead to their problems is somehow wrong.