presumably I want to play D&D with some form of horror mixed in, not a different game, so there certainly are limits to how far your subsystems can / should go.
I am also not sure it needs subsystems so much as narrative. A fear or insanity subsystem is always a bit suspect to me in D&D, you cannot tell the player to be afraid because their fear score crossed some threshold, at least imo
we also have plenty of horror genre monsters in the MM without them having different mechanics. Ravenloft just expands that list.
If there are any mechanics, it would be around the darklords I guess. That might be a decent idea, for ‘generic’ monsters I am less sure
would have to check 3e, as to Dark Sun, they could have just as easily kept it on the regular 2e ruleset. It made sense to refluff the player side (reframe races, add new ones, add new (sub)classes, adjust the item list), but beyond that (attribute range, …)? Anything beyond that can be handled in the narrative