So, some generic suggestions for increasing darkness:
a) So long as the players know the powers of the demon, can identify it, and know its weaknesses and limitations, it will be very hard to have the creature be horrific. As long as players are exploring territory which is known and understood on the meta level, they'll tend to be emotionally comfortable and feel safe. Mechanically, you can implement this by having each demon be unique on some level, randomizing appearance and powers and giving each a unique name. While true uniqueness requires more creativity than you can put into random tables, having each type of demon occur in a large number of variations will keep even players familiar with the rules guessing and uncomfortable.
b) Ideally, the players shouldn't even be able to clearly delineate what a demon is. Keeping in mind my above definitions of evil, one of the problems you run into is that players making choices regarding their behavior with respect to evil will tend to not make 'realistic' choices if they are aware of the realistic outcomes. It's much easier to be rational about a game than it is to be rational about the real world. Hearing that this is castle Anthrax, even a player not playing a character sworn to chastity is unlikely to avail themselves of the eight score attractive nubile young women in the obvious trap. I suggest that in real life, you'd be much more likely to be trying to rationalize that this in no way was a dangerous situation, and if it seemed unusual and weird that you shouldn't be looking a gift horse in the mouth, that a horrible death was completely out of the question, or that even if it turned out that it was a trap you'd be able to escape from it after taking just a little bit of pleasure.
One of the things that makes demons so dangerous in literature though is that the character don't realize the stakes (at least initially), and believe that they can always achieve the upper hand, or that demons are bound by certain laws or traditions they'll respect, or that they can be bargained with successfully, or controlled, or otherwise that the demon can be manipulated to their own profit. These are ideas of course that the demons themselves advance, because otherwise they'd be a good deal less seductive and dangerous. Demons advance all sorts of misinformation about their own nature in order to conceal just how terrible they really are, so lore in your world needs to be filled with wrong ideas that everyone knows to be true. Simultaneously you need to have demons be truly dark and working toward absolutely evil ends, and for many people in the game world to not truly believe that but instead believe demons are really just analogous to human villains with understandable motives. For example, people would believe that succubae can feel real affection and even at least erotic love, that demons admire and want to reward the unvirtuous, that people can obtain demonic power by performing evil acts, that demons are happy and even joyful, and generally that the payment is worth the cost. The truth is of course that affection, admiration, happiness and any other positive feeling whatsoever is completely beyond a demon, and that they only fake these things when doing so contributes to the triumph of evil.
c) It should be basically impossible to triumph over a demon in its own sphere of influence. Violence is pretty much useless against a wide spectrum of demons, and even when violence seems to work its not actually the violence that is harming the demon but the motive behind the violence (sincere compassion for the innocent, or genuine love for a comrade, for example). In general, near complete immunity to weapon damage is appropriate and golem like near magic immunity is likewise appropriate. As thoroughly wrecked and ruined creatures, making more holes in them doesn't generally work. They are beings that are all hole already. The only way to destroy them is to 'fill them in'. Evil is relatively helpless to oppose a demon, though of course humans are never so fully ruined as to be completely helpless. To do 'damage' to a demon generally involves empowering an attack on them in some way, for example, by confessing some truth you'd rather hide an attack on a deceiver demon might be effective that would be otherwise ineffective. By professing a reason you love someone else, an attack on a hate demon might be made more effective. A wrath demon can't be harmed in anger (sorry, barbarian), but only by someone completely calm and self-controlled.
Conversely, because they are often wholly ruined bodily creatures, most demons attacks are insubstantial physical danger, but extreme spiritual and moral danger. Demons generally need to act through others, or with the permission of others. They often act like debilitating parasites or diseases, eating away the vitality, rationality, and goodness of those they attack rather than making direct physical attacks. Like predatory fish with luminescent lures, they have sophisticated ways of disguising their nature and making being eaten alive sound attractive. They'll freely promise anything to get cooperation, lie without the slightest irony, and play on whatever weaknesses they find. Demons are also aware of their weaknesses. So for example, a demon that is only really threatened when someone acts to protect an innocent victim will prefer to first demonstrate that someone isn't innocent before directly threatening them, knowing that in this way they'll be able to gain inappropriate sympathy and render themselves invulnerable. Thus, they often will prefer to act subtly for some time, cajoling and exhorting people to evil, expose their evil, and then destroy them. Ideally they'll be able to portray themselves as some sort of at least quasi-force for good, destroying evil things and acting as spirits of rightful vengeance. Of course, those that avail themselves of this vengeance or condone it, have themselves taken the first steps toward their own guilt.
In general though, lucky is the person dealing with a demon that can dispatch it as easily as having a blessed weapon. Often even demons vulnerable to such things are only temporarily discorporated by the violence, and can reform themselves within hours or days of being so 'destroyed'. Like the Tarrasque, even if you are dealing with a lesser demon that you can fight off, some sort of ritual exorcism is often necessary to permanently destroy or banish such creatures. This ritual is often unknown and the demon will have arranged ahead of time to make the conditions for setting it up difficult. For example, it might require survivors to willingly sacrifice some advantage they have gained by violence or deceit, and the demon has arranged that at least some of the persons who survive at any time are too corrupt as to be likely to do so. Or some survivors, knowing that the demon will return if they do not do so, may have believed the lie that they have not been spared, but are truly liked and admired as the demons comrade and so will have greater and additional advantages if the demon does return.
d) Set up your gameworld so that these strategies basically work. Most dark stories are filled subtly peopled with highly flawed individuals who themselves aren't innocent and who make bad choices for vile if sometimes sympathetic reasons. They tend to involve anti-villains or anti-heroes as protagonists, or worlds where evil has basically won and so the power of good is simply ineffectual and incapable of strong resistance. If you have a world filled with truly virtuous people or any truly virtuous people with powers that allow them to resist evil - basically, if you have real paladins and saints - it's going to be hard to capture that vibe. Fortunately, in my experience its pretty rare for the PC's to occupy this position themselves, and as long as you can keep secret just how potent a truly saint-like character would be in a truly dark world, you'll have a willing line of Constantines and the like bumbling through the world never quite winning forever slipping on the sliding moral scale.
From a gaming perspective, much of this would be a lot easier to make gameable in a game like Pendragon that is concerned with virtue and quantifies it, than in a game like D&D which because of its war gaming roots has at the core of its gameplay the assumption that hitting things with a stick is fun and solves almost all problems. Another problem is the tendency for D&D to look at alignment as if it is in many ways basically symmetrical, differing only by the hats you wear and the physical tropes of the team members. If you went really dark, you'd break the symmetry. Of course, part of breaking that symmetry if you really want a dark world is having only demons exist and no corresponding powers of light.
I will say that D&D does follow this trope to at least some extent, as it's not unusual in published D&D worlds to see demonic invasions, kingdoms ruled by demons, and lots of demons just hanging out in the setting acting as captains of evil so that high level PC's have something to fight. It's rare to unknown to see the reverse, a published world with angelic beings bringing good to large portions of the world and threatening to render it forever benevolent, angelic beings ruling over nations in true benevolence and justice, and angels just hanging around the world ready to be helpful and actively squash any forces of evil that arise. Indeed, it's pretty rare to see a paladin or angel introduced into a story or setting without inevitably having that character fall. The rate at which NPC paladins or angels fall in stories is nearly 100%. That's a sort of darkness in and of itself. Note that the narrative this advances is that the best way to defeat evil is to be a bit morally grey and to compromise with it. Which, if you actually had demons that weren't just high HD bags of XP and loot, would actually be a pretty sophisticated strategy for disguising their nature and weaknesses.