I've spent some time lurking recently at RPG.net, and if that community's any indication, D&D as a whole is one of the least-well-regarded systems out there (at least from a mechanical standpoint).
I'll second this. Classic D&D in a non-D&D RPG group is about as well regarded as Windows at a Linux users convention. And for much the same mix of technical reasons and snobbery. (That said, it's still from what I can tell better regarded than e.g. Rifts).
4e on the other hand goes down pretty well with players who play non-D&D RPGs as a whole (see either RPG.net or Something Awful's Traditional Games board). This is because as far as I know, no other game does what 4e does as well as 4e, and of the things 4e doesn't do there is almost always a system that will do the game better than classic versions of D&D.
For example 4e doesn't really do absolute zero to hero. But with all due respect 3e and AD&D aren't much good at it either. AD&D wizards get to cast spells that won't blow up in their face and 2e fighters? Ouch. If I want to run a zero to hero campaign, I'm going to break out
Warhammer Fantasy Roleplay (either 3e or at least using the 2e magic rules - although even the 1e spell point system does better than D&D). It covers the "Starting at zero in a fantasy world that hates you" much better than a system with Gygaxo-Vancian Magic.
Classic (pre-4e but especially 2e and 3e) D&D has two real advantages - the first is that people already know it, and the second is that precisely because it doesn't do anything especially well it can do well
enough in a wide range of campaigns.
And I think maybe it's because the 4e / 3.x split finally put out in front of us, in the full daylight of blogs, forums, and chat rooms, something that we had maybe suspected but weren't really willing to admit to ourselves---That when it comes to D&D, rather than being "united" in our game of choice, we'd actually been demanding radically different things from E. Gary Gygax's magnum opus all along. The fact that it remained somewhat of the community's "lingua franca" for nearly 25 years is a testament to Gygax's original vision.
And to the embedded user advantage. Classic D&D is
good enough for a lot of things - so once you'd passed the hurdle of learning a complex rulesset (whether or not you regard this as an example of the Sunk Costs fallacy) you could use it for other things.
One reason the 4e / 3e split was so divisive, I think, is because when the 4e fans threw up their hands in joy and said, "FINALLY!!! CLASS BALANCE!!", all of us 3e fans went "Huh? Really? THIS is the game you wished you'd been playing for the past 25 years? Hmm. Didn't see that one coming." The concept that entire groups of players would so wholeheartedly embrace 4e's conventions seemed almost foreign to the 3.x-ers.....and the 4e-ers couldn't for the life of them figure out why the 3.x-ers couldn't see that the mechanical improvements were producing a "superior" style of game.
As a community we were forced to look across the table, across the room at our FLGS, and realize that what we assumed was a "shared D&D nationality" was more akin to groups of isolated city-states battling it out for territorial control. (I realize some of the more long-standing gamers probably came to that recognition long before 2008.)
The amusing thing here is that Gygax himself considered class balance to be very important. And introduced a
huge range of rules into D&D that are there to facilitate class balance. See, for example, the differing XP tables. Or the giving the fighters a free keep at level 9 or so when the wizards get a tower. Or even weapons doing bonus damage against large monsters and certain powerful spells having a serious danger of backlash. All things put into the game to assist with class balance. (And, not unsurprisingly, the bulk of the rules that the designers of 3.0 were kind enough to
remove from the game). 1e did not, of course, go far enough so they added weapon spec to the fighter, and added Cavaliers and (ack) Barbarians.
Because Gygax considered it important, and generally did a good job complete with effectively a
massive amount of playtesting, fixing things when they broke, people didn't have to worry about it so much. Gygax didn't allow Angel Summoner and BMX Bandit to happen - so the wildshaped bear with a bear companion summoning a swarm of bears to beat up the fighter was an issue that didn't arise.
As for 4e being what we'd wanted all along, perhaps, and perhaps not. It would
certainly have been possible to produce a game that satisfied the D&D players who love 4e without changing quite so much - but it would still probably have made wizard and CoDzilla lovers complain because the fighters were now about on a par with them and magic took substantial nerfs. 4e went further and harder in the direction it chose than it needed to. But the GenCon audience
cheered the removal of Vancian Magic with good reason. The 3.X primary spellcasters needed to be torn apart to satisfy the group that likes 4e. (For example the 3.X druid in 4e is literally three separate classes; the summoner, the nature loving healer with an animal companion, and the shapeshifter - and all are viable as classes, and the Cleric has been split into Cleric and Invoker.)
D&D Next will not be a "commercial" failure by any stretch. It will certainly be as profitable as 4e.
I question this assertion. Mostly because I'm not sure who D&D Next players are going to be. I don't see them being Pathfinder players - WotC is not as good as Paizo
at what Paizo does. I don't see them being 4e players - WotC is busy pissing all of us off. And I don't see them being people who stuck with AD&D - no published system is going to be a match for more than a dozen years of experience. I further don't really see D&D Next as a true OSR game for various reasons. And it certainly isn't an "appeal to the Indy crowd" game or a "let's change the language used from that of tabletop wargames to that of WoW because that's what people these days know" game.
I just don't see the market segment they are aiming at.
While I'll agree with most of your post, I'll quibble with this bit. I actively participate in rpg.net (under the same name, even) , and the opinions and trends I see there are not indicative of any other community I know, online or otherwise. It is a great place to discuss non-D&D games though.
I see rpg.net as being not very different from other places
I'd expect to discuss non-D&D games. And the opinions on 4e and previous editions of D&D as being not
that different from Something Awful's TG forum.