Barely hanging on the ledge of the 3rd floor (assume 20-foot drop) above a cobblestone street. A fall of that height wouldn't necessarily be fatal, but it would seriously hurt. Like "you're not dead but ain't going anywhere anytime soon" hurt.
I've made my peace with the abstraction of hp but as soon as the PCs are 2nd level, 2d6 damage is no longer a significant threat. I know it's not breaking the game but since we're looking for situations that should-be-scary-but-aren't, a 20-foot drop is not even remotely as threatening in the D&D world. Let alone a 30-foot fall or more.
I'm not convinced anything should be done about it; defenestration shouldn't always be a more effective solution than stabbing someone in the eye but on the other hand, it would be nice if a 10-foot pit trap was a little more than a mere inconvenience, and jumping down from the tower's roof wasn't just a more efficient way to climb down...