This isn’t how to run a spooky game but how to do so with the D20 system itself.
I want to run a horror campaign, and I feel like D&D is my only Real option because finding players for any other system is Tough. And horror is a small genre so I need all the chances I can.
But I feel that D&D is a poor system for horror because characters are so powerful. Even 1st level PCs have a lot of strength on their side, with spells and damage that can put a hurt on what they’re dealing with. Part of horror is feeling like you are at incredible risk, that you don’t have much of a chance of survival, but characters are hard to kill, and I think I’d be. To kill someone to demonstrate that fighting is risky as hell. It is the difference between dropping Joe Schmo into a horror movie vs trained soldiers. And then there s magic. Undead are just no as scary if you have a cleric, etc.
Also some mechanics feel clunky. The grapple rules for instance.
This isn’t how to run a spooky game but how to do so with the D20 system itself.
I want to run a horror campaign, and I feel like D&D is my only Real option because finding players for any other system is Tough. And horror is a small genre so I need all the chances I can.
But I feel that D&D is a poor system for horror because characters are so powerful. Even 1st level PCs have a lot of strength on their side, with spells and damage that can put a hurt on what they’re dealing with. Part of horror is feeling like you are at incredible risk, that you don’t have much of a chance of survival, but characters are hard to kill, and I think I’d be. To kill someone to demonstrate that fighting is risky as hell. It is the difference between dropping Joe Schmo into a horror movie vs trained soldiers. And then there s magic. Undead are just no as scary if you have a cleric, etc.
Also some mechanics feel clunky. The grapple rules for instance.
But back to the OP - I don't know if he wants D&D horror or modern horror or just anything horror, as long as it's a system that can handle it. There's lots of games that do horror really well, and D&D can do it, but I think you would need to keep it low level. But D&D would absolutely not be my first choice for really trying to evoke fear through mechanics. There are some alternate Horror and Sanity rules out there though, such as bundled with Sandy Peterson's 5e Mythos book.
Oh! Yeah, that's a 5th edition Cthulhu/D&D hybrid, that might work for the OP. That book is Bad. Ass.
Ghoul Island, a D&D horror campaign for Cthulhu Mythos 5e. Level 1-14. Now I don't see how you fully invoke horror with such high level characters, but it's worth looking at.
You're presuming horror has to come from combat when you talk player power. Horror can be twofold.
1. It's the realization that something you believed normal is abnormal, abhorrent, and for an added dash, you somewhat understand why it is that way. In the Curse of Strahd, the Abbott was making Strahd a stitched-together bride to undo his curse. Originally, in my game, he saw Ireena as competition and kindly asked the party to drown her. Because he was OF the Land, one of its favorite playthings, I treated him as undying (though they never attacked him). When the party talked him out of his madness, he said he'd seen the light from his god. The next time they met, Ireena didn't have a face. The Abbott sent his minions to take it from her, then healed her. The party had helped him see the light, that Ireena shouldn't die, and he solved the problem. Strahd was infatuated with Ireena, so he would give Strahd Ireena.
2. It's realization that you didn't have the control you thought you did. It's finding out the children you saved and returned to the Widow Flannigan went into her pastries. It's watching people do mad things no matter how hard you try.
Part of horror is feeling like you are at incredible risk, that you don’t have much of a chance of survival, but characters are hard to kill, and I think I’d be. To kill someone to demonstrate that fighting is risky as hell.
I caution you against relying too heavily on that notion. I subscribed to the same idea a couple of years ago when I ran Death House for my players, who love horror. At the end, the house is actively trying to kill the PCs by exuding smoke and whirring blades in the doorways. The players wanted to try some things to neutralize the hazards--putting cloths over their faces, jamming shields in the blades to stop them from working. I said those measures didn't work because I mistakenly thought that taking away the danger would kill the horror and suspense. All it did was make the players angry because I was shutting down their ideas. We had a long talk about it afterward, and I learned a lesson about running horror games in D&D.
I have horror aspects and arcs in my campaign all the time. I don't know why people think it's so difficult.
Being raised from the dead is either difficult or impossible in my campaign. To raise someone you literally have to retrieve their souls and track them down in the Shadowfell. There are things there that will be drawn to your life force like moths to a flame, so good luck.
Many demons and undead can be pretty horrific. Draining max HP, etc. I use the alternate rest rules so a short rest is overnight and a long rest is several days. You aren't going to get that long rest if you're trapped in a haunted house.
Throw in environmental factors that obscure vision or have a chance of making people sick (either exhaustion or poison). Magical darkness with creatures that have blindsight is always fun.
Don't be afraid to throw deadly encounters. If a party isn't properly challenged, just throw another wave of monsters. Have the penalty for losing an encounter be something other than death whether that's semi-permanent disability or some other ongoing penalty.
Replace a PC with a doppleganger and have the doppleganger played by the same person. Or, just for fun, take someone out of the room, asking them to bring their character sheet and a D20. Let them know absolutely nothing is wrong, but occasionally ask them to roll a D20 when you get back to the game table.
Borrow ideas from other games. D&D is incredibly easy to modify so if you don't like the grapple rules make something you do like.
I also tweak monsters a lot to fit the role, even visual changes can cause a visceral reaction. Ghoul like creatures have slime covered tongues that reach out and restrain the target, dragging them in closer. Add tentacles or bile or whatever you think of. Don't throw orcs, throw Orc Nurtured One of Yurtrus from Volo's guide.
Primarily though I find it being more about attitude and story than mechanics. Take a look at chasme demons for example. Have the party stumble across a bloated body (preferably an NPC they've dealt with before). The body starts to shift and move, only to rip open with the smell of rotting meat as large fly-like creatures crawl out. As they fly or crawl away they grow with sickening pops and crackles and grow the face of the individual they just hatched from.
Whether or not any of this works really comes down to how well you can set the stage, but also the players. Honestly, if they aren't into it no amount of skill or even a different game system will make a difference.
You're presuming horror has to come from combat when you talk player power. Horror can be twofold.
1. It's the realization that something you believed normal is abnormal, abhorrent, and for an added dash, you somewhat understand why it is that way. In the Curse of Strahd, the Abbott was making Strahd a stitched-together bride to undo his curse. Originally, in my game, he saw Ireena as competition and kindly asked the party to drown her. Because he was OF the Land, one of its favorite playthings, I treated him as undying (though they never attacked him). When the party talked him out of his madness, he said he'd seen the light from his god. The next time they met, Ireena didn't have a face. The Abbott sent his minions to take it from her, then healed her. The party had helped him see the light, that Ireena shouldn't die, and he solved the problem. Strahd was infatuated with Ireena, so he would give Strahd Ireena.
2. It's realization that you didn't have the control you thought you did. It's finding out the children you saved and returned to the Widow Flannigan went into her pastries. It's watching people do mad things no matter how hard you try.
Agreed, but the OP specifically mentioned the fact that his PCs are too heroic and don't fear for their lives. He/she wasn't asking about the broader menu of horror, but simply the fear of death.
Horror has been a part of D&D for a long time. Ravenloft is gothic horror campaign setting that is very popular.
Building tension is the key to a good horror game (and really, an action game too for that matter but to different effect).
Pacing in a horror game should be dependent on each session. At the beginning of each session the players aren't going to be scared. A DM has that time to ratchet up the horror.
The fear of the unknown is the greatest fear possible. So, don't show the monster(s) too early.
Players shouldn't feel like the game is "fair". Sandbox campaigns are perfect for this. Let the players know that threats are not engineered to their level.
Horror is not heroic. Provide circumstances where the players must choose between their PCs and innocents. Maybe they witness hags kidnapping a child into her lair, but do they dare approach? They know that the hag is much more powerful than they are. Maybe they approach with a plan in mind, but then there are 3 hags inside and they need to make the hard choice. Horror doesn't feel good.
(acts of heroism can occur in a horror game, they just come with dire consequences, thus the heroism. In standard D&D the expectation is that the PCs succeed at their 'daring' deeds routinely)
The goal in a horror game is often mere survival as well, rather than a heroic goal. In D&D the players could start with a noble goal only to have things turn and have survival be the primary one after they are in over their heads.
Ravenloft spoilers:
5e's Ravenloft adventure works because Strahd doesn't play fair. He knows the PCs, their capabilities, and where they are. He can walk through the walls of his castle, strike, and then walk away before they can react. He can do this over and over again. The only hope they have is to find the area of the castle where the final battle is destined to take place so that he cannot escape.
Play around with different pacing models too.
Here are a couple examples (The classic rising tension culminating in a final big threat common to D&D works in horror too):
The Vice: Threat level starts low and slowly continues, never stopping so that characters are worn down over time. They must fight against time to survive. Zombies are the classic example.
The Pit: Similar to The Vice only the party faces a huge threat that leaves them battered. They must then get to safety before their resources are fully depleted.
Finally, some house rules can be applied. Most common removing some spells. Ravenloft has done this over the editions. It is a demiplane without teleportation powers so that the PCs are trapped for example. Removing resting spells like Leomund's Hut would be a good idea.
Trying to throw deadly monsters to pc to achieve horror is a dead end.
you should try to aim the characters rather than the players.
a willing player may emulate horror and even amplify it,
