A problem that I often see in roleplaying games, and frankly, in D&D in particular, is that monsters completely fail to be scary. Rather, they are viewed as tactical game pieces and challenges. Arguably, this is what Gygax and Arneson wanted all along, but it is not something that I want, and I think the idea of reducing monsters to a statline that has tactical implications is to make them not monstrous at all.
Granted, it doesn't have to be like this. I've managed to engender some of the same kind of dramatic tension in D&D games (occasionally) as you get while reading a scary book or watching a scary movie... but frankly, not very often. Part of it is the paradigm and attitude that the players bring to the table; if they're playing D&D, then there's an assumption that they'll be facing challenges that they should be able to overcome if they're smart and tactically sound. It's just the tone and nature of the game, or at least it's often expected to be so.
I like monsters to be scary. I like players to really question whether or not they want to fight these monsters. But I admit that my success, what of it there has been anyway, in accomplishing this is something that I've done more intuitively rather than rationally, and I'm not 100% certain that I fully understand how to pull this off. So for this post, I'm going to noodle around some ideas and see where they go.
1) Monsters should be set pieces. The idea of going through a "dungeon" and fighting monster after monster has really diluted the concept. In any good story of supernatural horror, the monsters are never routine. Be careful about showing off too many monsters. Granted, D&D has truckloads of monsters. But don't assume that all of them are hanging around waiting to be discovered. Think of how the ancient Greeks did it. There were not "medusas"; there was one Medusa. There were not hydras, there was one Lyrnean hydra, etc. While you don't need to go to this extreme, keep in mind that monsters are much more monstrous if they are extraordinary. No matter how monstrous the description or the statline makes them appear, they never will be if they're routine. If you're not fighting monsters all the time, what are you having the PCs spend their time doing otherwise? Bad guys! Thugs, cultists, criminals, spies, and the like. Don't underestimate the value of contrasting monsters to larger hordes of plain old bad guys. Besides, bad guys are fun to fight for the most part on their own too. And also don't underestimate the challenge of dangerous animals. In this era of high powered hunting rifles, enclosed off-road vehicles, and completely tamed terrain, we forget exactly how dangerous it would be to come across a herd of wild elephants who feel that their calves are threatened out in the middle of nowhere with nothing to defend yourself with except a few sharp pieces of metal that you need to hold close to yourself to use. Or how dangerous a pack of wolves could be to a lonely traveler, or a pride of hungry lions.
2) Closely related to that, monsters should be unknown. There's few things more prosaic than the GM of a game casually announcing what his monsters are, when their properties, strengths and weaknesses are well known to all the players. Does this mean that you should only use unfamiliar and unusual monsters that you make up yourself or find in obscure third party sources? Of course not, but you should take steps to, again, keep your monsters from feeling routine. Think about possibly making them difficult to identify for a time. The PCs don't actually see them well until they're well into the thick of it, but they see the effects of their attacks on NPCs or something like that. Mix up your descriptions so that the PCs can't easily match them to a monster that they know. Statistics and descriptions can be decoupled and rearranged. One of my most memorable encounters was with a handful of hellhounds that I simply described very differently--I used some artwork from Paizo of Lovecraftian hounds of Tindalos to represent them, gave them a chittering Predator-like growl, and had their fiery breath transform into a vomit of tiny, toothy little demons that crawled all over their victims. Consider giving some of your monsters surprise abilities. A zombie that has a poisonous bite, or something. It doesn't have to be a big deal, just enough to keep the PCs guessing and unsure of what exactly it is that they're up against.
3) And closely related to that, make your monsters horrific. Granted, another of my most memorable scary monster encounters was with a little blighter that due to some relatively weak stats and some extraordinary good rolls by the PCs, ended up going down like a chump in the first or second round of combat, but that was the exception not the rule. He was creepy because of all the other stuff I had him do, but by and large a monster that the PCs aren't sure they can beat is one that's more likely to scare them than one that's only going to "reduce their resources by XY%" or something inane like that. In fact, ignoring the advice of the Challenge Ratings system completely (which is a good idea for a lot of reasons, only one of which I'll get into here and now), you can pit the PCs against foes which they literally can't defeat in straight-up combat. Monsters shouldn't often be creatures that cause heroic PCs to shout, "Huzzah!" and charge at to engage in melee; they should be monstrous. There should be a lot of doubt about how to deal with them, if they're up to the challenge, and what exactly can be done to get around the many strengths that monsters might have. Rather, you can have the PCs need to research specific dirty tricks or weaknesses that allow them to have a chance against the monster; without which they'd be committing certain suicide. A demon that can be banished back to its home dimension only via a desperately hurried ritual is scarier than one that can be banished back to its home dimension after the PCs just jump in and attack it.
4) And that gets a bit into my last tip; foreshadow your monsters. There's nothing worse than having a monster just pop out of nowhere, get defeated and then promptly forgotten. The chump monster that I mentioned above? He was mostly memorable because of the excellent (and extraordinary; I haven't quite had this level of success any other time, sadly) foreshadowing. While the PCs were attempting to sleep at a crowded and sleazy dockside inn, they were woken up when a young woman in the room next to them screamed. Barging into her room, they found that her eyes had been scooped out of her head. There was no sign of any attacker, just the sobbing victim's almost nonsensical cry that the last thing she saw was a hideous face over her shoulder in the mirror. A few more clues, a bit more foreshadowing... and then a half-glimpsed movement in their own mirror, and the players were keyed up, tense, and on edge. When it turned out that their invisible assailant was actually killed rather easily, it didn't diminish the feeling of dread and creepiness that the encounter had managed to elicit. And this foreshadowing hearkens back to my point #1; if monsters are set pieces, then you can craft an entire "adventure" around finding and defeating one, which means that you should have plenty of opportunities to foreshadow, to drop in unsettling or horrific clues or feelings, and generally ratchet up the tension on your way towards the final confrontation.
Anyway, like I said, much of that I do more instinctively rather than purposefully, and I have no illusions about the completeness of my methods.
