I'm not sure that horror is distinct here. Rules probably can't make me feel love, either, or deep sorrow. Does that mean that a dramatic (or melodramatic) RPG can't be done?
As I understand it, the point of rules in a RPG is to settle the content of the fiction. And fiction can make me feel (by imaginative projection, if not literally) horror, and love, and deep sorrow. So a RPG can do horror if its rules work in a certain way to support a certain sort of fiction.
Personally I wouldn't look to any version of WotC D&D to do this - the rules complexity is, I think, too apt to intrude on the establishment of those sorts of feelings and these games don't have other devices to keep those feelings front and centre. But maybe BW could do horror - it is complex too but does have devices to keep feelings at the forefront. And something like Wuthering Heights or Cthulhu Dark could manage it too, I think.