Inventing RPGs was an achievement. But the achievements - in innovation, conception, technique - didn't stop in the 1970s or 1980s. Modern designers have solved problems, and created possibilities, that had eluded earlier designers.
It isn't necessarily linear development. There are branches. Some innovations are just innovations and not solution. Some old designs work fine. The idea that whatever design elements you prefer are some sort of evolutionary advantage over ones you don't is myopic.
Who asserted linearity, let alone "evolutionary advantage"? As
@soviet said, you seem to be projecting.
Gygax's design works fine if you want to play White Plume Mountain, and if you are prepared to supplement your rulebooks with an ethos and play culture that those rulebooks presuppose but don't articulate (for instance, the play of those classic adventures depends on players understanding that certain architectural features of the shared fiction - doors, corridor directions, the possibility of hidden egresses/rooms/pits/etc - are central to play in a way that certain sartorial features of that fiction - the cut and colour of NPC's clothes - typically are not).
But Gygax's design is not very good if you want to play something that will, in play, have emotional resonance something like A Wizard of Earthsea, or John Boorman's Excalibur, or even a typical REH Conan story. It is the mismatch between a particular design, and a desired play experience, that gives rise to a
problem.
Hickman solved the problem using one particular technique: massive GM force over the content and direction of play, such that player decision-making about what actions to declare for their PCs becomes a secondary concern at best in determining how things unfold. Many tables have found this satisfactory, and continue to find it satisfactory. But not everyone does. The Hickman approach centres melee combat as the main site of play where player action declaration may tend to make a difference; and in doing so also creates a well-known source of possible dissatisfaction, when the PCs TPK in a way that is emotionally/thematically unsatisfactory. For some people, therefore, a
problem remains.
But these problems have since been solved. And further solutions continue to take place: in another current thread I made a post identifying how "neo-trad" RPG design solves problems that arise in "trad" design and in "indie" design (the problems are different in each case).
When it comes to Monsterhearts vs HERO, there are play experiences that Monsterhearts is delivering that HERO is not. And of course vice-versa - no one who wants a play experience of moment-by-moment tracking of the kinetics of combat is going to choose Monsterhearts for that.
But that isn't really a system thing, is it? That's just the style of play and the social contract at the table. I mean, you can have mechanics (metacurrency or whatever) but you can also just play in a fashion that lets players define elements and make choices about what happens.
Meta-currency, though beloved by ENworld posters as a "narrative mechanic", is largely a red herring.
Apocalypse World is player-driven RPGing, but has no meta-currency.
What is key is more fundamental aspects of play dynamics. Like,
where do the elements that are used to frame scenes come from?,
who decides what is at stake in a scene?,
how are the consequences of resolution established?, etc.
Now one answer to all of the above is
the players and GM agree on things, freeform style. Which is fine in itself, but (i) raises the question of why are we even bothering with mechanics? - the stuff on our PC sheets is just descriptors that guides our freeform, and (ii) means that we will not get outcomes that come as a surprise to everyone, or that no one would choose if given the option. (Points (i) and (ii) are not original to me: Vincent Baker makes them here:
anyway: Rules vs Vigorous Creative Agreement.)
Assuming that you want, in play, outcomes that are sometimes ones no one would choose - like, say, the PC running across the golf course startled by the sprinklers - then you are going to want mechanics and associated procedures. And assuming that the fictional activity that produces these outcomes is going to be something other than exploring Gygaxian dungeons and skirmish-level combat, then you are going to need mechanics and procedures different from those that Gygax invented and that many RPGs have emulated.
@Campbell has given the example of strings, and
@AbdulAlhazred has talked more generally about the PbtA play structure. I'll give an example from the play of Torchbearer 2e:
I described the Moathouse, and they decided they had three options: frontal assault, stealth infiltration, or trickery. They decided to go for trickery: they were emissaries from Roy (Megloss's bandit underling
whom they had driven off at the Tower of Stars), seeking to establish an alliance of Lords of the North, banding together for greater security and profit. Fea-bella equipped her half-truths and evasions (improvised weapon); Golin equipped the fact that he was already allied with the Dire Wolf (+1D defence, I decided); and Telemere equipped a "prop", namely, his knowledge of the stars and omens that revealed that the tie is right!
The PCs approached the gates and announced themselves, and a bandit underling ran of and fetched their leader. But even though the bandits were all without "weapons" (and so -1D) and had to use Beginner's Luck Lore Master for their Manoeuvres and Defence, it all went badly for the PCs (which will happen when you are all Injured (-1D) and two of you are Sick on top of that (-1 further D) and one of you can't help because Afraid). Their disposition of 4 was eliminated while the bandits had lost only 2 hit points of their starting 6, and so owed only a minor compromise.
The bandits' response to the PCs' lies had been incredulity, and an insistence that they surrender. Which, having lost the conflict, they did.
The compromise I suggested, which the players accepted, was that as they were marched off to the dungeons, the bandits would not realise that the Dire Wolf was the PCs' ally - so they have a Wolf on the inside!
This conflict tripped over the Grind again, and so all were Hungry and Thirsty.
I described the PCs being marched up the steps into the ruined great hall, then down the stairs into the dungeon. When the bandits instructed them to stick to the side of the stairs, Telemere (who instinct is to check to see if he is being watched when entering somewhere new) made a Scout check (on the Grind, given that he was Exhausted) and saw the Creeping Ooze clinging to the wall and ceiling on the other side of the stairs.
From the stairs, they were led through the old, ruined storeroom past the cells, through the Gnolls' chamber and into the captives' room, where they were given some water (recovery from Hungry and Thirsty) but stripped of most of their gear, and locked away.
It is basically impossible for AD&D to produce this sort of outcome in any systematic way. There is no procedure, other than perhaps just a roll of the reaction die, to determine whether or not the Moathouse bandits are tricked by the PCs' story about having been sent as emissaries by Roy. And if the PCs fail, there is no way to establish, as a binding element of the fiction, that
the PCs are led away and imprisoned and
that nothing more happens to the PCs are a result of the players' loss than that they are led away and imprisoned. AD&D doesn't bind the players - so at any moment the players, being led away, are free to declare that they make an attack or try to run. And nor does it bind the GM - so if the players have their PCs go along with the bandits, they have no basis for confidence that the GM is not going to escalate the stakes at any moment.
Rolemaster has slightly better ways of resolving the trickery (via the influence and interaction table) but it can't really treat it as a group conflict in the way Torchbearer 2e does. And it has exactly the same problem that AD&D does, of having now way to transition from
the failure to trick to the
being led away and imprisoned without both players and GM having authority to escalate the stakes at any moment.
In AD&D and RM, most of the time the only
binding consequence that precludes escalation by one or other participant is
death. Hence why combat becomes foregrounded as a site of conflict in both the fiction and the mechanics.
Torchbearer is a B/X-influenced descendant of Burning Wheel, and BW uses a similar (not identical) approach to social conflict - the Duel of Wits. The similar conflict resolution, but with les of a B/X ethos around the fiction, is how, as I posted upthread
I have played BW where a debate between two travelling companions about whether or not one will repair the armour of the other matters - in that it is high stakes in play, it can be resolved in a way that is not predetermined and does not just depend on the two players talking it out, and it feeds through to the game's system for recognising and rewarding significant episodes of play.
Monsterhearts "strings" mechanic, as described by
@Campbell upthread, seems like a descendant of Apocalypse World's Seduce/Manipulate move:
When you try to seduce or manipulate someone, tell them what you want and roll+hot.
For NPCs: on a hit, they ask you to promise something first, and do it if you promise. On a 10+, whether you keep your promise is up to you, later. On a 7–9, they need some concrete assurance right now.
For PCs: on a 10+, both. On a 7–9, choose 1:
• if they do it, they mark experience
• if they refuse, it’s acting under fire
What they do then is up to them.
Focusing on the PC aspect: In the fiction, the character feels a "tug" to do this thing; in the real world, the player feels the lure of an XP. Or, in the fiction, the character feels bad or conflicted about, or at least upset by, saying "no", and it affects their focus and judgement; in the real world, the player has to make a roll to find out what happens as their PC (metaphorically) acts under fire. Perhaps both (if the roll was 10+).
This isn't the same method as BW or TB2e - in the latter the consequences are about stakes and how the subsequent scenes are framed; whereas in the PbtA games the consequences are more "local" and emulate the feelings of the PC and the way that affects their decision-making.
But both methods make gameplay possible that simply cannot be done in games like AD&D or RM or, I believe despite my lesser degree of experience with it, HERO.
I don't just think player agency is the best part of TTRPGs (as opposed to other games) I think it is the whole point.
But the GM framing a scene in which my character, having failed to trick the bandits and hence having had to surrender to their superior force, has been led away and is now locked up, does not
reduce my agency.
How did I get there? Because I staked my PC's liberty on a chance to trick the bandits and hence infiltrate their Moathouse; and I failed.
What can I do now, in this new scene? Declare whatever actions are permissible given the fictional framing of being locked in a small dark cell, and given the genre conceits of the game we're playing.
So framing it in terms of agency I think is a red herring. It's really about
what does the game permit to be staked, and
what scenes does it permit to be framed. AD&D and RM have limitations here, that BW and TB2e do not. Of course if you don't want a game in which my PC's liberty, as opposed to their life, can be staked, then these other games may not be for you - but that's about preference for stakes and scenes of a certain type, not a preference for agency.
I don't think roleplaying requires "acting" at all. You, the player, don't need to.act angry, you need to have your character respond angrily with as much or as little playacting as you are comfortable with. I just don't care much for immersion or inhabiting a character.
Inhabitation of character is central to my experience as a player. As a GM, I don't inhabit, but I do want to feel the emotional crests and troughs, see the elation at victory and grind my heel into the players' faces when their PCs lose. And not just when their PCs die (grinding one's heel in such circumstances is in fact a little cruel).
The inability of the Gygaxian design to support this is a problem (for me). It has since been solved, and in ways more satisfactory than the RMC III Depression crit table.