Li Shenron
Legend
I don't want a lot of combat, honestly I don't want the players to fight much. That's sort of the point of this thread, because the standard mindset of a D&D player is "If it's there I can kill it" and generally PCs in D&D will kill it. From my experience, unless something is ridiculously higher level, the PCs can take it out.
...
The focus is avoid, escape, survive not stand and fight. The kind of campaign I'm looking for is very much like the 1e days in that it emphasizes exploration and is dangerous. It'd be about encountering nightmarish sights, then running away from them or finding a story way to win, as opposed to attack and damage.
Sounds great to me.
So let me reframe my core question:
If I must use D&D because that's what everyone plays, I will to run a game using the D&D ruleset, not a D&D game with horror elements. How can I avoid all those rules and make them feel powerless in a game where they have tons of power?
Guys.
I don't need tips on running horror. I know how to DM it.
My problem is that the rules get in my way. I don't want the players thinking about hit points and attack bonuses.
The problem is that most D&D players expect the game to be A LOT about combat, and as you say combat is really not compatible with horror. In addition, players expect a good chance of their PC not to die, in fact they don't want to, even if they want to believe that their PC doesn't die because they are "good" at the game (ha!). Even more so, the most recent editions are clearly grounded in the idea that it should be fast to recover from combat, because players want... more combat and quickly.
I don't think there is a way to avoid combat rules unless you avoid combat, period.
Even then, you still have a potentially serious problem with many spells that work outside combat, for example if they can spoil mysteries/secrets, or if they can provide too easily an effective protection from harm. And of course, healing... any harm that simply translates to regular hit points is useless to deliver a sense of fear or horror, but there is anyway a spell for any other kind of harm, unless you change the rules yourself.
Eventually, this makes all fighting-oriented characters, magician characters and healing characters get in the way, because their player obviously will want to use their PCs abilities. What can you do? If you let them play the usual classes, they will assume they have to build a good Fighter or Cleric, only to found out later that the DM has introduced measures to make them ineffective. If you tell them not to play classes that get in the way, what characters are left really, Rogues?
The point is that D&D is designed for the characters to win, and by definition "horror" is based on the characters to lose. Yes I know, the trope is usually that ONE MAIN character or two "win" (i.e. survive) at the end of the story, but at a devastasting cost of usually losing everyone else they care about and their life as it was, meaning that the horror lingers on in the consequences... just not possible in a game designed like D&D.
The general ineffectiveness of the characters is essential in true horror stories. Even the survivor finds an unconventional, often serendipitous way to "win" at the end. Movies where the protagonist wades through hordes of monsters are parodies of horror, or simply action movies with increased gore imagery.
So I am afraid you are going to have to resort on more tips and tricks, and get a somewhat satisfactory result but not as good as in a book or a movie. Maybe you can find a way to deliver that feeling of powerlessness in a way around the game. First of all, if you forget about running an extended (or open-ended) horror campaign and just run a SHORT adventure, potentially even single-evening, then you CAN kill off the PCs with no regard. You can also consider multiple PCs per player, have a very large number of PCs (such as 4 players x 3 PCs = 12 total), and start killing them off to deliver the sense that the risk is real, without cutting off any single player from the game?