IME the two primary ways to terrify the Players is the known and the unknown.
Many players know the various types of monsters or abilities that are REALLY bad. Dragons, Giants, Fiends, are all generally avoided at lower levels, because the players know that they are far too weak to face such a powerful foe. Pack Tactics is an ability that also terrifies my players, since they know that getting surrounded is almost certain death just from the sheer number of attacks with advantage. Some negative effects have been neutered in 5E, so I've houseruled them back to being more evil (for example, Life Drain lasts until the Rest/Recuperate downtime activity is taken, not just a long rest). The players are aware of this, making monsters more fearsome once again.
The unknown is always something a player should fear (except newbs and the foolish, who quickly learn better). Something I've done for personal preference has become a major issue for my players: I use the OD&D, BECMI, and AD&D descriptions of monsters, not the more recent ones. Because of this, players don't always know what they're facing, and this has lead to some dramatic deaths (such as a level 1 PC surrounded by kobolds). Even a troll may not be obvious, and because of this, they may not know to use fire against it. In addition, monsters react to things, even if they don't have a vulnerability to it (anything with sunlight sensitivity shies away from radiant damage, for example), giving some false impressions. By limiting the players knowledge, I've inadvertently made several monsters more terrifying.