I feel that you're using incredibly narrow definition of a story.
Perhaps. Nevertheless it's been very important in the history of RPG design and play, both for better or worse (eg consider the 4e DMG advice to
skip to the fun). I think it's at least plausible to suggest that right at this moment the most influential approaches to RPGing that are neither D&D nor D&D-adjacent are PbtA and Fate, and both of these approaches to RPGing have a concern with
story in at least roughly the way I have characterised it - recognisably dramatic/thematic concerns producing a crisis/climax and then a resolution.
One can see it making its first published appearances in TSR material in the DL modules (eg the "obscure death" rule); in the original Ravenloft (the antagonist has recognisable, dramatically resonant motives); and really coming to the fore in the advice found in the 2nd ed AD&D core books.
Some stories are descriptions of real events, and thus cannot follow any specific structure, albeit the narrator may emphasise certain elements to create dramatic resonance (story is not just sequence of events, it is also how those events are described.) Story that doesn't follow a certain pattern doesn't stop being a story; it may be a boring or bad story, or perhaps it could be refreshingly surprising one. I feel the latter is what many of us want from RPGs.
I don't really feel the force of your contrast between typical story structure and "refreshingly surprising" story, and frankly I think the story structure of (say) a serious play through of Tomb of Horrors will be getting closer to Warhol's Sleep or Empire than to anything refreshingly surprising!
But in any event I stand by my assertion that not all RPGing is concerned with story in my sense, but a great deal is. And if you want it, getting it
may come into tension with the technical play of the system. It's precisely this possibility that motivates the DL "obscure death" rule, for instance.
I do believe that story--or at least narrative--emerges from play, yes, but I think it's (at least mostly) only visible looking backward.
A typical 4e combat plays very much like a lengthy remake of a fight scene from Rocky or The Karate Kid or Zorro - the conflict breaks out, the protagonists find themselves set back/on the ropes/under pressure, then they dig deep into their reserves, sometimes literally get their second wind, and prevail.
This is not an accident. It's a feature of design, and we can point to the elements that support it - eg NPCs/creatures have most of their survivability front-loaded into hit points and have a good chunk of their offensive power loaded into at-wills (this also has the benefit of reducing the cognitive load of running them); PCs have a good chunk of their survivability located in healing surges that need unlocking through deploying resources and/or making moves, and have a good chunk of their offensive power loaded into limited-use abilities that require clever judgement to deploy well.
And it's not visible only living backwards. At least in my experience, it is felt on the way through.
A good 4e skill challenge will produce the same experience, though the system is less finely tuned here than in respect of combat and so is a little bit less reliable.
Moving beyond D&D variants, there are whole schools of RPG design dedicated to ensuring that the story is visible not just looking backward but in the moment of play. That's why they go under the label
story now. My favourites are Burning Wheel (intense) and Prince Valiant (light); the most widely played - I think - is PbtA.
There's a difference between playing a game to go through a pre-planned story, and playing to generate a narrative.
A further difference is
playing to experience a story in my sense, where that story (i) is not pre-planned and (ii) will emerge without anyone having to curate it.
In my experience, playing AD&D will struggle to deliver this without fairly heavy curation. (The version that gets closest, in my experience, is the original OA.) Given the close resemblance of 5e to AD&D for these particular purposes, I'm prepared to assert, with some but not total tentativeness, that the same is true of it.
Burning Wheel and Prince Valiant are different in this respect. And not by coincidence - by design.
Talking about foreshadowing in a TRPG seems kinda like trying to pound a square peg into a round hole.
Yet it is widely discussed as a GMing technique. And the cover of I6 Ravenloft depicts Strahd brooding on his castle parapet.
Part of the rationale for the soft move/hard move approach of PbtA is to achieve the same effect as foreshadowing, but without the need to curate in respect of it.
If one is using "curation" that way, I think a GM who chooses the types of instigating events to frame into the fiction is plausibly curating the fiction, or the narrative, in such a way as to achieve a particular presentation or effect--but maybe without concern as to the result
<snip>
I think that's at the heart of the conflict in the OP, which I don't see: It's taking as a given that one can or should reproduce literary effects in a TRPG. Since I think it's not possible--and in fact I think it's a bad idea to try--because the media are so different, the conflict in the OP pretty much literally does not exist for me as a GM.
OK. But I think it would be silly to deny that no RPG designer or player has ever been concerned with it. The history of TSR publications from the DL modules through 2nd ed AD&D through 5e adventure paths shows that the biggest commercial publisher of RPGs thinks that achieving story in the sense that I am using it, which is the same as in the sense of
@Manbearcat's "storytelling imperative", is an important thing.
The existence and comparative success of PbtA and Fate as non-D&D systems shows that there are players of non-D&D RPGs who think that story is an important thing.
The tension to which the OP draws attention arises out of a particular clash, that can occur in some RPGs but not all due to details of system and technique, between
the logic of technically skillful game play and
the desire for story as more than just a sequence of fictional events.
I'd say that if the rules for a given game ensure a literary structure will emerge from play without the people at the table doing anything to make it happen (other than play the game by the rules) then the rules are curating the fiction to generate a story.
I think that would be a largely uhelpful thing to say. It's would be like saying that tossing a coin instead of making a decision is "letting the coin, or the rules of the coin-toss, make the decision". As a casual metaphor that's harmless enough; but of course the truth is that a coin-toss is an alternative to making a choice, not a particular mode of deciding (hence why we have integrity commissions for lotteries and casinos!).
The whole point of "story now" systems and the techniques that they rely on is to allow the story to emerge in the experience of play (not just in retrospect)
without anyone having to think about it or curate it.