But how much difference does it make? D&D has a frightened condition, but it's just mechanical. What kind of mechanic can you implement that would affect the player?
Because (and maybe I'm just totally missing the boat) so far other games work "better" for horror because people identify more with their PCs. If a player is not going to get into that mindset, I have a hard time imagining how game rules or lack therein is going to matter.
I don’t think it’s a matter of having a mechanic to reinforce the characters’ fear, but rather having a mechanical
framework that facilitates making the player fear for their character (and not
necessarily for the character’s life, though that can be one way to do it.)
There’s this Horror RPG called Dread. Its core mechanic is a legally-distinct-from-jenga block tower, which is used in place of dice for action resolution. When you declare an action that involves an element of risk, the GM can ask you to take a block from the tower. If the tower remains standing, you succeed. If the tower falls, your character dies. And that’s basically it, there’s not much more to the game mechanically than that. The DM narrates a scenario, the players roleplay their characters, and when asked to do so, they pull blocks from the tower.
At first, it’s pretty low-risk - you do a thing, you pull a block, you move on. but as the game goes on, the tension builds and builds as every move you make leaves the tower a little more unstable. Every successful action makes the next action more likely to be the one that knocks it down. You go into the game expecting your character probably won’t survive the session, so it’s not like you’re particularly attached to the character. But the game’s core mechanic takes the narrative tension and makes it viscerally, physically real. You can’t help but feel the rush of adrenaline pulling another block from that already barely-standing tower, and that sensation gets you in the right mood for the cathartic horror experience.
That’s just one example of a game with a mechanical framework that is built to facilitate horror, but I think it demonstrates that what’s needed to make horror work is a dissolution of the barrier between the character’s emotional state and the player’s emotional state. If you’re just describing your character being scared or rolling dice with some penalty or other to represent the character’s fear, you’re not getting the catharsis that horror is meant to deliver. You have to feel the character’s fear, and for that to work you need a game system that facilitates getting the player into the character’s head space. That’s part of why a lot of horror games are really big on props.