When you say "nor should they", I assume you mean something like "nor should they if they are going to yield a game that I prefer".
Because there is no objective RPG-design reason why rules should be as you characterise them ("a framework for operationalising what the character want to do based on their own internal logic"). For instance, in 4e a STR paladin can have an at-will attack called Valiant Smite. This grants a +1 to hit per adjacent enemy. And it is not simply an operationalisation of what the PC wants to do based on the PC's own internal logic: it is a metagame ability which ensures that
the paladin who has it will be valiant, smiting his/her foes when they surround him/her. Inspiring Word, a Warlord power that allows an ally to spend a healing surge and thereby recover hit points, is another metagame power in 4e. And it does not simply operationalise what the warlord PC wants to do based on their own internal logic. It also allows the player of the recipient of the power to narrate his/her PC's inspired recovery from shock and pain, and resurgence into the fray!
Plenty of other RPGs also have mechancis that do more than simply operationalise what the characters want to do based on their own internal logic. For instance, in The Riddle of Steel or HeroWars/Quest, a character's attacks will be mechanically more potent if the target of the attack is someone whom s/he is statted up as hating, or if the attack is made in the course of defending someone whom s/he is statted up as loving. 4e doesn't have these sorts of emtional mechanics, but it has stuff in the neighbourhood - radiant attacks, for instance, make divine PCs more capable when confronting the undead, those violators of the gods' ordained order.
Dark Sun doesn't seem that idiosyncratic to me: it's seems a pretty standard sword and sorcery setting. Bizarre desert cities with mad wizard tyrants and oppressed slaves are hardly out of the ordinary for pulp tropes.
Its peculiar use of the D&D demihumans is a bit more distinctive, but that distinctiveness only arises because so much D&D material has been quite conservative in its handling of them.
Dragonlance strikes me as even more stock-standard: gruff dwarves, graceful elves, valiant knights riding metallic dragons, the whole works!
But the Red Box isn't Tolkien-esque at all, except in the flavour of its demihumans.
D&D orcs aren't Tolkien-esque. Tolkien has no distinct goblins, hobgoblins, bugbears, gnolls, regenerating trolls, thouls, etc. Tolkien's wizards don't cast fireball at all, and a fortiori don't wipe out rooms of goblins using magic - Gandalf (if a magic-user at all, rather than a cleric) uses a sword; and Elrond is both an elf and a master healer (though, in Red Box terms, presumably not a cleric).
And the Red Box has basically no material to support a Tolkien-esque play experience - I'm not saying it would be impossible to play a fantasy romance using the Red Box, but I reckon it would be pretty hard.
The way you phrase this tends to imply that you are pointing to some desires of WotC, and some features of 4e - whereas I think you are really only conveying a biographical fact about yourself.
After all, for me (and, by his own testimony, for @
AbdulAlhazred ) 4e captures the look, feel and (desired) play of D&D better than previous rulesets. Here is the sort of RP experience that Moldvay Basic - my first encounter with D&D - promised:
I was busy rescuing the captured maiden when the dragon showed up. Fifty feet of scaled terror glared down at us with smoldering red eyes. Tendrils of smoke drifted out from between fangs larger than daggers. The dragon blocked the only exit from the cave. . .
I unwrapped the sword which the mysterious cleric had given me. The sword was golden-tinted steel. Its hilt was set with a rainbow collection of precious gems. I shouted my battle cry and charged.
My charge caught the dragon by surprise. Its titanic jaws snapped shut inches from my face. I swung the golden sword with both arms. The swordblade bit into the dragon's neck and continued through to the other side. With an earth-shaking crash, the dragon dropped dead at my feet. The magic sword had saved my life and ended the reign of the dragon-tyrant. The countryside was freed and I could return a hero.
My play and GMing of D&D has been, to a significant an extent, an attempt to find a mechanical way to make this sort of action - which is quite different from dungeon crawling and looting, and much closer to the Tolkien-esque - the centre of my play.
I've used AD&D (especially Oriental Adventures). I've used Rolemaster. And now I use 4e. For me, the story elements haven't changed in any fundamental way - the single most prominent source of story material for my Rolemaster play was AD&D modules, for instance. But the mechanics have. They've done a better job of delivering my desired play experience.