Sure, I can totally agree that that's not the focus of every game, but...
Setting is always going to inform my character's existence, who they are and why they are, and I'll admit I struggle to picture a game where the setting doesn't have some impact on the nature what's present and what's possible in the course of pursuing action, and thus have an effect on the character that is being unearthed.
And I wouldn't call either of these things irrelevant, or unimportant. It might not be the linchpin, but that will still be an important factor in how I approach those situations. Background color is, honestly, another not inaccurate way of describing what I was talking about above, the stuff that helps me inhabit a character so that I can make honest and true decisions that reflect their reality.
Can I ask what you're picturing when you thinking of a "setting [that] might be mostly irrelevant to that"?
Let me paint you a picture of one such type of scenario.
You're a 14 year old in pre-state Utah in a Wild West That Never Was. You're being initiated as one of "God's Watchdogs." You will go out amongst the Towns, amongst The King of Life's flock. You will perform ceremony. You will deliver mail. You will arbitrate disputes. You will mete out justice.
But that is later. You're here. Now.
Are you hoping that you accomplish something during initiation? Maybe you exorcised a demon? Maybe you showed yourself to be a crack shot? Maybe you exhibited knowledge of scripture and doctrine that set you apart from your peers?
Great. You're you. I'm the terrible demon. I'm the breech-loading rifle's tendency to recoil violently and the winds tendency to foil. I'm the inscrutable doctrine and your peers' scrutiny.
Or are you looking for growth, learning, a changing of troublesome habits?
Ok. Now you're playing your character's foul temper that you have a tendency to succumb to or the fear of galloping at full speed in your horse's saddle. I play your teacher in their attempt to disabuse you of your worse nature.
We determine the pivotal moment of such a situation, and we take up our respective dice pools. You roll your relevant Stats and later (or not) bring in your Traits, Belongings, and Relationships as they become (or not) applicable. I roll 4d6 + 4d10. We launch into conflict that is akin to poker.
We Raise and we See those Raises as we say what we're doing and we select dice to represent that. We Take a Blow (when we can't See with 2 or less dice) or we Reverse a Blow (when we can See with only 1 die).
By the end of it, your character has a new d6 Trait which represents the outcome of this conflict. You also almost surely have Fallout which is short term (eg change a Relationship die size to d4 for the next conflict) or long term change/impact to your character (eg change the die size of an existing Trait to d4 or remove a Belonging).
Regardless, what happens in that scene isn't informed by high resolution setting material. Everything is very nascent. We don't need to know the name of your teacher. We don't need to know the name of the Steward of Red River Rock or if Red River Rock is even a Town. We don't need to know if an oil baron or the railroad via the Fed is trying to move in on the territory. We don't need to know your relationship to the Sin of False Doctrine (although maybe we'll learn about it if its related to the coming conflict or the stuff you bring into it via PC build and the conflict resolution mechanics). We don't need to know much. What we need to know we know. You have a temper that gets the best of you and an aspiration to defeat it. You've got Stats, Relationships, Traits, Belongings and a situation in front of you. We'll find out more about who you are, maybe more about one or more of those Relationships or Traits if they come into play and what you say about them when you use them. And we'll find out "what's what" in the end (and your character will change as a result).
That is what I'm talking about. A situation pregnant with aspiration, emotion, immediacy and a character with means and baggage. And a resolution (mechanically and the consequential fiction that accompanies it).