Maybe because both of those games adhere to a design philosophy, instead of to a marketing philosophy?
Because of it's massive size, D&D is in a bit of a unique situation. It is best served by being "good enough" for a broad swath of people, especially those new to RPGs in general, rather than laser-focused on a single play experience. It always is going to have to serve many masters -- it is BETTER when it does, because of the diversity of the audience and the game's status as the standard-bearer for the hobby. A D&D with a more focused design (like 4e tried to have) is worse at being the standard-bearer, worse at appealing to a broad audience, and those things make it worse at being "D&D."
This means that D&D is probably always going to be in a situation where it's going to be a little unfulfilling. Storytellers will be annoyed at the combat grind, strategists will be irked that the combat is designed to move fast rather than reward a tactical depth, grognards will be annoyed at the lack of dungeoneering, etc., etc., etc. A broad game is going to be designed to serve most of these folks "well enough," within certain weightings. A more specific game is always going to be able to do any of these things with more fidelity to a coherent design philosophy than D&D will. Those games don't need to somehow unite three to five different play goals under a single system.
Balance is not important at all. I don't think it improves the play experience one bit. I realize that may be unpopular, but it is based on extensive play and there is no compelling evidence or proof to the contrary.
I have a slightly more nuanced take: Balance is subjective. For instance:
What the hell are you talking about tho? Some of the most miserable game experiences of my life have come from broken d20 builds making combat basically a one-man show. Like you don't need to experience it to understand how soul draining it is to play a game of Gary Stu and his retainers... surely someone who regularly plays TTRPGs has enough imagination to understand why that would suck.
This is objectively a lousy game experience. But at a table that didn't care much about combat? Or that had players who were good at spotlight-sharing? Or at a table where challenge dungeons were the norm and being a combat god wouldn't protect you against failing your save vs. that deadly trap? Or a table where PC's had very defined "roles" and the combat god was useful in their niche but where being a good face or a good trap-finder or a good scout was just as important? A different context changes that lousy game experience into just a forgettable one, or even a "fine" one.
Like, the game was originally designed with one class (the Fighter) doing the fighting and the rest worry more about other elements of the dungeon. Combat balance wasn't always a part of this thing, and it still isn't for a lot of people.
Combat balance in terms of not letting one character dominate is
probably important for a lot of tables today, so it's something the game should probably take into account, but it's also debatable. And every design decision has a cost.
