I steer clear of Call of Cthulu becaus the system is built to do things that don't interest me: 1) Everyone will die or go insane, it's inevitable. 2) It's all about sanity and I find mental illness depressing, not scary. 3) It's very INVESTIGATION heavy. While I like the imagery and body horror, Lovecraftian stuff doesn't interest me.
Okay let me try to be clearer.
I. Do. Not. Like. D&D. I loathe it in my soul. However. It is hard to find players willing to play anything else, and it is hard to find a group who wants to play horror. Therefore I feel the options are D&D or not finding a group for months (or, as suggested in other threads I've started asking how to find players for non-D&D rpgs, "Go play D&D until you can convince the people you play with to play another system").
If I must play D&D, then I have to find a way to crowbar the game I want into D&D. Were this 3e I'd say "I want to use the d20 system to run horror where everyone is playing weak NPC", but I feel like if I say "d20" that will turn people off because it's "not D&D". If there are not fighters and clerics then it's too far off the reservation for players.
I don't want a lot of combat, honestly I don't want the players to fight much. That's sort of the point of this thread, because the standard mindset of a D&D player is "If it's there I can kill it" and generally PCs in D&D will kill it. From my experience, unless something is ridiculously higher level, the PCs can take it out. And the same things just don't work; even if Michael Myers is 10 levels higher and has an Assassinate ability, the players are not going to split up to be picked off one by one and a guy with a knife is simply not menacing to a group of five swordsmen, it's just a hack-a-thon.
Let me give you an example. The characters are in a room with a mirror. They can see in the mirror a monster among them, but only in the mirror. A D&D player says "How do I attack it, what's it's AC?" and a player in an RPG where they aren't a hero says "I NEED TO GET OUT OF HERE" because whatever that is they have no idea how to deal with it and they are certain whatever it does will ruin them. D&D gives a ton of tools to deal with it though, so the players think they can deal with it with the tools they have.
The focus is avoid, escape, survive not stand and fight. The kind of campaign I'm looking for is very much like the 1e days in that it emphasizes exploration and is dangerous. It'd be about encountering nightmarish sights, then running away from them or finding a story way to win, as opposed to attack and damage.
So let me reframe my core question:
If I must use D&D because that's what everyone plays, I will to run a game using the D&D ruleset, not a D&D game with horror elements. How can I avoid all those rules and make them feel powerless in a game where they have tons of power?