I've had good luck with both the familiar and the unfamiliar. In a lot of cases, it depends on the situation with your group. If you have a very imaginative and immersed group, then having the bank teller suddenly rake her fingernails across your wrist and leave a bloody trail of scratches that itch intensely and ooze ink instead of blood can be terrifying. If you don't, though, they're going to go "huh, ink-blood, huh? That's probably poisoned. How much damage did it do? Are her fingernails artificially long?"
On the other hand, if you've got a more crunch-minded group that is very aware of the game-like-nature of what you're doing, then sending in a monster that they know has a death-effect gaze, a slam that bestows negative energy levels, and the ability to shoot a stream of ichor for 5d6 acid damage is gonna be darn scary. A more immersed group will probably react gamely to such a monster, roleplaying their fear, but the players will most likely be going "Okay, it's a monster, I'll act afraid and then, when my character gets desperate enough, he'll Double-Tap on his first attack and move to be 40 feet away from it, out of gaze-attack range."
My group has a mix, which is unfortunate. A couple of players are perfectly capable of freaking out when I describe a hallucination of snakes crawling out of the eye sockets of a murder victim, while a couple others react with "Oh, yeah, snakes, okay, I back away. That's not good."
The one thing that I've noticed seems to work for everyone is something I'm getting better at: Destruction or deformation of the body. When a hallucination or a crime scene involves someone who has been eviscerated, with their guts spread around them in a circle, tied spread-eagled to the four corners of the room with their own tendons, and with every tooth in their mouth pulled out and shoved into their now-oozing eye sockets, both the immersed people and the game people get worried.
On the other hand, if you've got a more crunch-minded group that is very aware of the game-like-nature of what you're doing, then sending in a monster that they know has a death-effect gaze, a slam that bestows negative energy levels, and the ability to shoot a stream of ichor for 5d6 acid damage is gonna be darn scary. A more immersed group will probably react gamely to such a monster, roleplaying their fear, but the players will most likely be going "Okay, it's a monster, I'll act afraid and then, when my character gets desperate enough, he'll Double-Tap on his first attack and move to be 40 feet away from it, out of gaze-attack range."
My group has a mix, which is unfortunate. A couple of players are perfectly capable of freaking out when I describe a hallucination of snakes crawling out of the eye sockets of a murder victim, while a couple others react with "Oh, yeah, snakes, okay, I back away. That's not good."
The one thing that I've noticed seems to work for everyone is something I'm getting better at: Destruction or deformation of the body. When a hallucination or a crime scene involves someone who has been eviscerated, with their guts spread around them in a circle, tied spread-eagled to the four corners of the room with their own tendons, and with every tooth in their mouth pulled out and shoved into their now-oozing eye sockets, both the immersed people and the game people get worried.