If you want to introduce horror into D&D, you have to include the Sanity Score. This allows the DM to use various thriller/horror tropes with actual mechanical effects. As someone else said, you can do horrible things to a character, but it's hard to scare a player. Sadly, one of the most terrifying things I recall in D&D was a critical hit chart during 2E, because dismemberment was very common, (including the head). Having a character die sucks, but having your character crippled horrifies a lot of players.
Call of Cuthulu would like a word.
I'm a huge fan of horror movies (which sadly never scare me, but I love them anyway), and the creature feature is a sub-set of horror. You also have slasher, gothic, supernatural, black/dark comedy, and found footage (ugh). They often overlap with other genres, as was pointed out the Alien franchise is sci-fi horror, but Aliens also dabbles a lot into action. The slasher can mix with mystery and thriller with an unknown killer. Technically Found Footage shouldn't be a sub-set of horror, as you can do the same technique with other genres, but I'll admit I find it distasteful and unappealing in any form.