You can absolutely do pure horror with D&D 5E. You just have to
- start with fairly low level characters who have not "seen everything, done everything."
- limit availability of magic and spell casting. If everybody is capable of the strange and uncanny (i.e. powerful magic) then you're setting yourself up for a fall. I use a world where magic is illegal, full of witchhunters, so showy spells are a risky business, and superstitious NPCs are hard to trust. Horror thrives on the normal becoming abnormal. A sparkly WoW setting won't work.
- introduce the insanity rules and really pay attention to fatigue and fright.
- I use a house rule that isolated loner characters take a penalty to sanity and fright checks because they feel more vulnerable. Then I isolate them by events.
- Tweak the healing and rest rules to be harder, using something like the Gritty options in the DMG (I use a house rule that is somewhere between gritty and normal).
- build a believable, supportive, relatively normal world to start off in. This will contrast with the strange and isolated world the character later find themselves in.
- Make them genuinely like some NPCs. Have various friendly NPCs snatched away gruesomely and mysteriously. Have secret societies hinted at, so the players don't know who to trust.
- design reasons why each character can't just walk away from this (revenge, monster has a crucial item or piece of a larger puzzle, no escape)
- Design your locations: take time to build atmosphere and character to each. Add fog, rain and darkness. Small tight tunnels. Deeper and deeper places. Signs of increasing madness and degeneration.
- Design early scenes where a foreboding event happens. A gored body is found, or people start going missing. Escalate gradually. Don't reveal the enemy for as long as possible.
- Never name the monsters, only describe them. A cunning, stealthy bugbear is actually pretty terrifying if you don't know exactly what it is. (This enriches D&D games generally). Add unexpected abilities.
- Design the monster’s actions to ensure the players really hate it. Take away a loved one or a generous contact. Do something cruel to them.
- When the players close in on the monster, around 2/3rds of the way into the adventure, design a massive spanner in their plans, turn everything upside down and give them a suprise twist. "It's not here! But that can only mean it's... (cue screams in the far distance)"
- hit the players hard and be prepared to kill. They need to plan hard to survive this, and they can't just walk away from it.
- Get the right music and eerie sound effects: you need
- wind
- wind and storm with thunder
- forlorn inn
- the theme from John Carpenter’s The Thing
- Kickass combat music (Terminator 2 or Batman Begins - Molossus are a good place to start)
- Dripping / chittering cave tunnels
- If someone starts joking around let them… for a little while. Stay only gently amused, and keep the mood tense by keeping things moving. If necessary, punish persistent jokers with cruel monster interventions. Make them get vocal about their hate for the enemy instead.
I try and do this for all my D&D games, because the players start to care more. We are returning to 'trad' dungeon delves with 5E and finding them to be changed places. I always wanted to be immersed in the game, not just joke my way through it, and I'm finding the rules system has pretty much all I need.