I am not really trying to defend anything. I have come to a realization during these discussions that a TTRPG game can't do horror. Rules can't make you feel horror. I didn't expect to come to this realization, it is just where the conversation took me.
That's a reasonable position and not one I'd argue against per se, and it's one I've heard espoused on and off since the 1990s.
Personally I sympathize with that position, but I think you can create a situation where it's a lot
easier for the DM and players to mutually create a kind of viable horror atmosphere, especially if you're going for certain types of horror - gothic horror and body horror for example are, imho, a lot easier to make work than slasher horror, because it's way too easy for the latter to turn into Scream, and that only works as horror in a sort of one-shot.
I think the one big mechanic type that virtually all horror could be helped by is mechanics that stop the PCs being able to act in completely rational and sensible ways.
This is the biggest bar to to horror in TT RPGs in my experience. The players and how they run the PCs. Even cooperative players who say they want horror, tend to run their characters in an extremely level-headed and practical way. It's like every single character is basically Ripley in Alien. If every character in Alien was Ripley, no-one would have got the facehugger on their face in the first place, and even if they had, they wouldn't have been let back in the ship, or would have been hard-quarantined for a long period (possibly the entire rest of the journey), and what's more the person not being let back on or being quarantined would have been totally understanding and reasonable about it.
CoC often works as horror despite this, not because of sanity, but because of relative PC helplessness. Even if the players are running the PCs are totally level-headed, the utter helplessness against a lot of mythos creatures undermines that, and panic can set in. Often it just doesn't though. I've CoC groups do stuff like calmly run a last stand against some monster as it kills them one by one, or everyone decides to hold off the monster so one PC can get away, even though, if they RPing to the hilt, that might not be what would happen. The thing is though, it's easy to find an excuse to be rational in these situations, and if you don't have any mechanics to break that up, it's going to happen a lot.
If was looking for one mechanic to promote a horror vibe it would thus be some kind of panic mechanic. Something that, some of the time, took control of your PC away from you temporarily, or let you act, but only within constraints. This is not a mechanic it would be hard to retrofit to D&D, either, but you'd need to recalibrate combat expectations and the like based on it. You'd also want it to broad enough that it could apply out of combat. Ideally the behaviour would relate to some personality concept chosen by the player.
Reading VRGtR I noticed they had some nice stuff with motivations for characters in a horror settings (mostly cleverly unhelpful motivations), and I think there are some inspirations there for the personality concepts. Some characters would be prone to fleeing, others freezing, others hiding, others determinedly repeating the same actions, however fruitless, and so on. I think you'd want such a mechanic to have more nuance and personality than typical fear mechanics, which just force PCs to flee or suffer penalties. The point overall, would be to break up the plan the players had, and force the to improvise and potentially panic a bit more (obviously some of this can also be done on the monster design side).
Another thing I've been considering is some sort of "survivor bounty", i.e. whichever PC or PCs survive, they get some kind of edge the other PCs don't, next time, maybe make it granular, like there are four conditions you need to avoid to get the "full benefit" (half-health, downed, er... I'd have to think about the other two), and these conditions would basically be at least somewhat antithetical to calm team-play. Make it so players are incentivized to have their PCs act in a more survival-oriented way, and a slightly less cooperative one.
Oh man actually god, maybe Survivor the TV series has an idea here, maybe in the horror campaign only one of you can be the "Final Girl" or Sole Survivor or w/e. You're sort of gamifying horror here but what you're trying to do is simulate that "will to survive" that PCs often gladly abandon in favour of teamwork and so on. Kind of abandoning that whole "everyone is a winner" tenet of RPGs, but maybe that's part of it. Hmmmmm. This would need a lot more thought.
