I wouldn't say there's been any need to "unlearn". At least not in any concrete way.For example, a D&D player walks into a Numenera game. Numenera has stat pools which can look like hit points to a D&D player (but they're not...). Part of the fun in Numenera is expending effort (which reduces one's stat pools) which permits special abilities and increased luck. The D&D viewer might not see this though, because from her perspective, expending effort = reducing hit points = killing your character.
<snip>
Did the switch to any of the above games involve unlearning things from prior games? I'm guessing your group isn't the type to say, "I don't want to play this, because this other game does it better," but there might have been some "I don't understand this rule, because it's handled clearly in this other game."
There is the need to get the hang of the dynamics or rhythms of play in some cases. Not for really light systems like Cthuhlhu Dark or Wuthering Heights, where the mechanics are absolutely transparent on the most cursory inspection; but some systems are more intricate.
For Cortex+ Heroic/MHRP a couple of sessions help the players get the hang of what they can do to build their dice pools, and maybe more importantly for me (as GM) helped me get the feel for how to use the Doom Pool effectively.
For Prince Valiant, a session of play helped me get the feel for where the maths of the system fall - it's very simple dice pool resolution (either opposed or against a target number of successes), so the maths isn't hard in any way, but it still helps just to get a sense of the rhythm of things. One asepct of this is that (unlike, say, D&D combat) all failure consequences are freely established by the GM within pretty lose parameters, and so part of getting a sense of the rhythm is getting a sense of what sorts of consequences to narrate.
The particular example you give - of someone not wanting to pay their pool because they're "killing their PC" - isn't something I would worry about with my group. They will look at a system and pretty quickly work out what the player-side resources are and how to spend them. In that sense I would say they have a high degree of "game mechanics literacy".
It would never occur to me to think of d20 Modern as ground-breaking. I'm otherwise not sure which games of the ones I mentioned you think are D&D with different resolution systems.By traditional, I'm usually referring to the design sense rather than temporal. So, for me d20 Modern is pretty traditional, along with many other systems that some think are ground breaking. My personal experience has been that, even with some of the systems you mention, the results are pretty much D&D with different resolution systems. I'm confident that varies from group to group.
<snip>
most games have (like D&D) extensive rules and subsystems for combat resolution, but little more than lip-service to non-combat fictional resolution (often without any subsystems whatever).
Rolemaster has non-combat resolution systems - more details below. Classic Traveller has many non-combat resolution systems, and I can report from recent play experience that they are robust, with the exception of the system for on-world exploration which is weak, and a disappointing contrast given the strength of the other systems.
RM is famous for the rich fiction of its crit charts, yes. The relevance, to me, of this aspect of RM (which extends well beyond its crit charts) is that it establishes a strong correlation between mechanical resolution and consequences in the fiction. So RM (at least in my experience) never produces outcomes like I hit the orc for 3 hp. How's it doing? We always know what's happening in the fiction - eg the orc is bleeing badly from its arm and has dropped its sword.Not too familiar with RM. Is that the one with the tables for combat results that includes super-detailed things like..."toe chopped off"? If so, I'll just say...ahem, and leave it at that. I think that authorially-minded players and GMs can act this way in almost any system. My crux of the issue for me is how well-supported such things are within the mechanics.
In my experience, this makes a difference to play. For instance, the death rate is far lower than in D&D because NPCs retreat or surrender. Players can have their PCs offer quarter (You're hurt. Surrender now and you won't die!) Etc. In the first RM game I GMed, which has an AD&D-like rate of level gain, the group's main fighter - a paladin - never killed a foe in combat until 5th level.
The same sort of thing is true in non-combat. We never have I bluff the guard! Woohoo - a 26 check result! We always know what the character is saying, and the social resolution charts tell us how the guard responds. To give an example, the Near Success result says "Keep talking, your audience is becoming more friendly. Modify your next roll by +20 [on a d100 check]." The player has to tell us more stuff that his/her PC says. We know that time is passing. We know that the guard is listening but not persuaded. It's not as robust a resolution system as a 4e skill challenge, but I've never encountered a hint of Diplomancy in nearly 20 years of RM play. (RM is much weaker when it gets to journey resolution. It's not a coincidence that this is a partial point of overlap with Traveller.)
And I've had a lot of surprises, as a GM, in RM play because the results are binding on the GM. The maths is nowhere near as tight as PbtA games, but the players have bonues on their PC sheets and dice in their hands, and if they want to they can read the Influence and Interaction resolution chart.
I thiink we've had this conversation before. I'm a big believer in the reality of what Ron Edwards calls "vanilla narrativism". It's mostly what I play. Think about Prince Valiant, widely acclaimed as one of the first and best narrativist systems: it doesn't have anything like compels. Nor does Burning Wheel (to point to a fairly well-known contemporary system). Nor does Cortex+ Heroic/MHRP. Nor does Apocalypse World.If that ends up contributing to the play-narrative...sure. However, that ends up being totally up to the DM, doesn't it? I don't recall anything like Fate's compels, where you might get a Fate point if the kidnapped victim is one of your family or something. (Which you can then spend later to push the fiction in a direction you like.)
All of those systems rely on the GM to do the framing, and have as the most basic player-side activity declaring an action that responds to the fiction the GM has presented. In BW and Cortex+ Heroic that can include actions that establish new story elements (eg as Assets in Cortex+; using Wises and the like in BW) whereas that's much less a part of Prince Valiant or AW. But establishing those new story elements is not the core player function even in BW or Cortex+.
AW and Cortex+ have their own (different) approaches to how the GM should frame which I'll set aside for the rest of this post. Both Prince Valiant and Burning Wheel rely on the GM to frame things that are interesting and speak to the players' evinced concerns for their PCs. This won't work if the players don't evince conerns, nor if the players won't actually play their PCs and engage the fiction. That second turtling issue is a pretty well-known one: you don't need fate points or compels to deal with it, but rather (in my view) vivid fiction, sound mechanics and deft adjudication of conseequences. Provided that the GM is framing things properly, the player doesn't need to spend fate points to push the fiction in a direction s/he likes - s/he just declares actions for his/her PC.
But framing things properly brings us back to the first issue, of evinced concerns. This issue is (in my view) most easily tackled through PC build, which is where I see a fundamental change between classic D&D and something like Oriental Adventures. OA PCs have build elements - families, masters and mentors etc - which can actually matter in play. The contrast can be drawn with Gygax's DMG: this tells us that a 1st level PC wizard was taught by a master of at least 6th level, but there is not the least hint that that master will actually figure in play. Whereas OA absolutely presents the PCs' background and loyalties as material to build the game from.
My own view is that if a GM doesn't want to follow player leads and/or if a player doesn't want to establish and engage fiction that will provide and act on those leads then giving this a mechanical overlay via fate points won't help much. If a group is ignorant of the possibility of this sort of play then I can see how the mechanical framework can draw it to their attention. But I think the key thing is the approach to play, not the mechanical framework.