That is, if you like your play to represent actions in a self-consistent fictional world, if you want your play to simulate reasonable events in a self-consistent world that you've built, then the system matters.
Only for people who think those self-consistent fictional worlds are produced by the game mechanics.
There are other folks, like me and my group, which honestly enjoy consistent, detailed fictional settings, but don't see them as the product of the formal rule set. Settings are a kind of fiction, and they're brought to life the same way fiction is: through narrative, description ,etc.
My settings are supported not through rules, but through conversations w/the players and a series of shared assumptions (the 1st of which is, "even though the world is fantastical, most things work the same as here").
It matters because you want those rules to allow actions that are possible and disallow actions that are not possible.
In standard D&D, it is impossible to generate a specific injury -ie to poke out an eye. If you use the combat rules to extrapolate a general "physics of the world", then specific injuries are
impossible. No one-eyed pirates or grizzled one-armed veterans. I find this, well, silly.
Therefore, I don't treat the rules as physics. Because it leads to a less believable fictional world.
It's not accurate to suggest gamers who don't see the rules-as-physics also don't care about story or about the exploration of detailed, consistent imaginary worlds. Which is to say, there's more than one way to simulate a world (ie, a way that doesn't involve algorithms which consistently yield nutty results).