I truly think the fact that 4E works so well for you is a happy coincidence.
They trumpeted things like "ease of DMing" over and over in 2008. They didn't sell the ideas that you praise.
their primary goal didn't work, and they lost a lot of existing fans for their trouble. But they also made a specific niche of fans very happy. I'm not sure if you are an outlier from that niche or just a corner case within it. But either way it clearly worked out great for you.
Whereas I don't think it's a coincidence. Which isn't to say they set out to please me. But I don't think it's a coincidence that a game designed to be an attractive newbies game, where people would sit around having fun pretending to be elves, is good for me. What would such a game involve?
I'd start with limited operational play (check), because it's tedious (check), and limiting it will facilitate robust scene framing (check), so that the challenges confronting the players via their PCs are clear and easily engaged (check), with action resolution that will produce satisfactory pacing for those challenges built in (check) so that people will have fun pretending to be elves (check).
And I'd clearly frame the role of the GM as one of building the challenges (check) but leaving it up to the players to choose how they'll engage them (check) and also leaving it up to the players to build the PCs they want to use as their vehicles to engage those challenges (check). Because magic items are part of PC build, players will be mostly in charge of them too (check).
This "player empowerment" will mean that players want to set their own goals/quests (check), so we'd better make on-the-fly encounter eyeballing and action resolution pretty easy (mostly check - battlemaps can be hard to draw up on the spur of the moment, but maybe we'll sell some dungeon tiles).
Where did they drop the ball? Well, (i) the maths is a bit dodgy, both in combat, out of combat (DC revisions), and in the fact that the two don't mix as well as they should; and (ii) power bloat and especially feat bloat is so obviously built into the system maybe something should have thought from the start about how to regulate and/or contain it.
And (iii) there are also some infelicities in the way player and GM interaction is set upA, despite the fairly clear attempt to delineate roles: players are in charge of central story elements (player-chosen quests, paragon paths, epic destinies) but the GM is expected to set up the relevant encounters (eg destiny quests) and no guidelines or machinery is given to facilitate this.
And (iv) the advice on how to run skill challenges is criminally inadequate, given the rules text for HeroWars/Quest and Maelstrom Storytelling that they could have drawn upon. Related to this, (v) the useful information about the role of keywords in establishing fictional positioning is buried in a DMG discussion of affecting objects, whereas it should have been central to page 42.
Still, I think they did a pretty good job, and as I said I don't think it's coincidence that their conception of a game for new roleplayers also supports the sort of vanilla narrativism that I enjoy. All that it takes is to take the colour that the game empowers the players to bring in (via the story elements over which they have control) and beef that up a bit, and to compensate from my own know-how for the lack of GM advice on how to respond to this stuff.
What makes me think that they may be capable of squaring the circle with D&Dn is that 4e squares a circle to some extent. Played exactly as written, I think 4e is probably incoherent, because of that missing advice to the GM and players on how the distribution of authority over story content and scene framing is meant to work. But drifting to a workable solution is trivial. You either downplay the fictional content, leave it primarily in the hands of the GM and treat it as mere colour when the players introduce it, and you have the notorious "4e as boardgame". If you ignore the keywords on powers and abilities, you can even check fictional positioning (and page 42 with it) at the door if you like. Alternatively, you drift in something like the way I do, and you get coherent vanilla narrativism. And I'm sure there are other functional driftings possible as well. The squared circle.
What's missing from the game, from the point of view of a stereotypical author stance, immersion-oriented simulationist or Gygaxian gamist player? I'm probably not the best person to answer this, but my answer would be this: 4e lack an expectation that players will treat with the fiction only via their PCs' ingame actions, and furthermore it has mechanics that give the players a chance to shape the fiction at the metagame level (and a lot of the time isn't even very coy about them).
The challenge for D&Dn, that makes it harder (in my view) than 4e is that it is precisely committed to bringing back in this sort of expectation, while at the same time making 4e play - however drifted - viable.
I'm not worried that D&Dn won't let me inject theme into play. Any RPG can do that (although mechanical alignment rules, if they exist, generally have to be purged), because it's mostly just a question of story elements. So even if my 4e game is unusual in its narrativist focus, that's not a big deal on its own. It's the mechanical features of 4e that make it driftable to a range of non-sim games that are the big deal. For me, they're the features that let me
keep theme, rather than (say) strategy, at the forefront of play. Those same features let a 4e boardgamer playing Lair Assault keep combat tactics, rather than operations and strategy, at the forefront of play.
So I don't think it's just me who is getting more from 4e than easy GMing.