In my experience, the thing that most makes a monster scary is that the party doesn't know what they are dealing with or what it can do. It's easy to terrify inexperienced players. Once they have the monster manual memorized and can recognize monsters from clues in their behavior, the saving throws they are provoking, etc. monsters tend to just be different tactical problems. You can enhance this by homebrewing monsters, modifying existing monsters, or reskinning monsters so that it isn't obvious what the players are facing.
Monsters that present truly tense tactical problems tend to also be the ones that put you into situations that 'suck'. The PC's are always trying to produce a situation where winning is a matter of mathematical inevitability, and that bad results are improbable to the point of being impossible. If monsters don't have advantages to leverage, the PC's will almost always succeed in that because they are flexible and tend to have multiple advantages they can lever in either range, mobility, defenses, or the action economy. Particularly at higher levels with good spellcaster support, PC's can tend to find answers. So you have to have monsters that can find answers as well and force reactions from the players.
In short, any monster that can force the players out of their comfort zone tactically, and make them really dig for answers is "terrifying". If the combat plays out differently than normal, then the combat will be tense and the monster will earn respect. But if the combat just is a matter of making the normal choices and/or hard slapping down the monster with a spell that exaggerates their weakness, then the combat will not be 'terrifying'.
Achieving that is part of what makes for good encounter design.
There is a tension here. Part of what makes a swarm terrifying is that they are immune to weapon damage. But immune to weapon damage risks being not fun, because for a lot of characters there isn't a lot they can meaningfully do. So really the only question you might be asking here is, "Do you have arcane casters?", which isn't that fun of a question and isn't really that interesting of a combat. Hence, scenario design with swarms is particularly tricky, in that you need to telegraph them a bit and put them in a terrain where there is something meaningful to do if you aren't dropping fireballs.
Likewise, monsters that can do long term damage to a character or their equipment that isn't easy to recover from are terrifying, but heavy reliance on that can leave players feeling more depressed and annoyed than they are immersed in the game. So again, telegraph and give options to avoid the nasty situation commiserate with the skill level and experience of your players.
My point was using monsters to break important equipment - like destroy spellfoci of various casters thereby limiting their spellcasting potential, crushing the shields of sword and board warriors, smashing the bows of archers...etc
