Following up: and I think this can be very frustrating to folks who are looking at this from a design perspective, because the behaviour seems illogical. If the point of the game is to produce a great shared storytelling experience, then the design of D&D could be (and has been) improved in so many ways. But I don't think that's really what the game is about. Or not all of it - not even close.
Or, you know, I'm totally wrong and just spinning in circles. One of my problems is that I tend to think like a scientist (e.g. observe a phenomenon and try to hypothesize) but I'm trained in the humanities (with a bit of biology) and my math kind of sucks. Maybe I married a scientist to try to make up for my shortcomings.
You're running up against a phenomenon that is conflating several different goals of play... some incompatible with each other.
For some, it's a seeking of some fantasy desire to be fulfilled, a "What would I do if...."
For some, it's a game to be won or lost.
For some, it's a game of pushing your luck.
For some it's a way of resolving items not agreed to.§
For some, it's merely inspiration for freeform collective storytelling.
For some, it's still miniatures wargaming in campaign mode - fights and just enough narrative to get from one to the next.
§: this is actually several things, as it can be mechanicalizing ...
- Narrative control (who gets to decide the thing)
- Success
- quality of success/failure
- cost of success
- effect of failure
- Who gets "screen time"...
Now, I love me boxes to put things in... but some storygames go well away from the core tropes of RPGs, such as exclusive character control and character scale, and the lack of individual winner.
Once Upon A Time is a game that generates a story when played. But it has no personal ownership of any characters in story. It has no success/failure determination. It does have pure narrative control, basically by hijacking the narration when an item for which you have a card is mentioned. First one out wins.
Aye, Dark Overlord is a game of telling why you failed as a minion. It lacks the continuity of most RPGs, but you're going to hear several stories in a single play. Each with a narration (ideally in character), and hazards thrown in by others.
Hobbit Tales from the Green Dragon Inn is another storygame that isn't an RPG; the play is similar to ADO... but it's tolkien themed. Too bad it's out of print.
There are others, too... they take the game element.... and the story element, but are they Roleplaying? ADO and HT are in character... and
The Extraordinary Adventures of Baron Munchausen... they're close, but one doesn't hear of TEAoBM as a campaign game much... and one seldom hears of D&D as a Push Your Luck dungeon crawler with new characters every session... but both have been used for those oddities by unusual people.
There's no single taxonomy that makes the RPG market into a "There is as clear single winner" situation; different games are aiming at different play-goals. Not-D&D is successful by being Not D&D, by doing other things.
And some, by doing some D&D things better than D&D while not doing other things that D&D does okay at.