Yeah, I think it makes for good gaming if characters to occasionally face threats they can't handle, but it is tricky. Often players are just relying on some sort of metagame thinking to know they are outclassed, which is not ideal (either knowing a monster's CR is above their current capabilities, or realizing when the DM starts going into great detail about the piles of bones outside the monster's cave that means they need to run).
A couple of techniques that I think work:
MUCH more powerful monsters: With adult dragons, beholders, demon lords, etc. it should be very clear very quickly that most parties are outgunned. With lesser threats it is harder for players to realize they are overmatched until too late.
Superior numbers of known foes: If the party has a hard fight against 5 ogres, tell them that 20 ogres are coming down the road. If they are not smart enough to run then, they deserve whatever happens next.
Waves of enemies: For every enemy the party drops two more join the fight - the party should realize that if it does not flee it will get overwhelmed.
Invulnerable enemies: Let the players know that their attacks don't seem to be having any effect. This works well for low level parties without magic weapons, but you can also create monsters that require specific weapons or attack types to defeat. This can be used to set up the "we need to find dragonglass weapons in order to kill these guys..." type plot lines.
Redshirts: Let the players see the monster chop through a bunch of NPCs so they realize how tough it is.