Range: right off the bat, HP is MUCH better 120' range, then affects a 30' cube. Fear is self and then a 30' cone. Plus if your flanked, fear will only get one side unless you can get lucky on positioning.
The range is better on HP, agreed. The AOE is awash, while HP is larger it is also more difficult to position without hitting allies.
Flanked is bad for both. Yes you can technically hit flanking enemies with HP but then you hit yourself as well since you are between them. If you fail your save the spell fails completely as you lose concentration on it. With either spell I think the way you are normally going to deal with flanking enemies is move take an AOO and then hit them both.
While the targets have to retreat they can still attack both the caster and other party members
No they can't. They have to take the "dash action", so they can not take the attack action or any other action except dash. They can use bonus actions and reactions but the options to attack with those are very, very limited.
Plus if the target loses sight of the caster - it gets a save.
To be clear it gets a save at the end of the turn if out of LOS, not when it breaks LOS. At that point they have moved 2X movement away from the caster. If they manage this and make the save on the first turn they will need to take dash again on the next turn just to get back to where they were before it was cast and if they are humanoids they come back into the fight without their weapons, wands, shields or whatever else they were holding. If they don't save they keep running at twice movement until they do, even after line of sight is broken. Unless they hit a dead end, this takes them further and further from the fight every turn they are affected.
You also do have some control over where they run because they have to move away as long as there is somewhere to go and because of the frightened condition they can't move closer to you. So you can position yourself to force "away" to be in a certain direction. For example you open the door to a dead end room and cast fear into it. If you then move to the other side of the room, they have to run through the door you entered, even if it means going by all the other party members, because it is the only direction to go. On the other hand if you do not want them going through the door and want to trap them you can stand between them and the door and they can't approach it because they have the frightened condition and can't move closer to you. If you have something like misty step you can act like a herding dog and make them run back and forth in one direction then the other, running by all your party members and sucking up AOOs every turn. It just really depends on how open it is. A hallway or dungeon you have a lot of control, and full control if you can get to the other side of them on your turn. In an open area you have limited control, you can basically keep them from running in one direction, unless your move is high enough to herd them in a certain direction.
This means what it's cast, the party can then focus fire on (ideally) one opponent at a time, this is huge. Even if multiple targets save, you still only have to deal with the ones that do, that's as good control as it gets.
Not really. First, they can use an action to wake them up and if the incapacitated enemy goes after or at the same time as the awake enemy the guy that was charmed does not even lose a single turn. If you fail to get at least half of them, it will often only result in a single lost action for the ones who MADE their save as they use their action to wake their buddies who go on to attack.
Second if the caster loses concentration they all wake up immediately. If the caster loses concentration in Fear they can all turn around and attack (without their weapons) but often they are hundreds of feet away by that time and will need one or more turns just to get back to where the fight is happening.
The problem I have run into with HP is you need to get all of them for it to be really effective for more than a turn or two. Even if there is just one enemy that saves; he uses his action to wake his buddy who goes next in initiative, his buddy is no longer incapacitated and uses his action to wake the next guy, he uses his action to wake the next guy ...... In 1 round they are all back in the fight and as long as just ONE enemy saves they can do this unless movement restrictions prevent it.
With Fear it is usually going to affect those who fail for a long time, often for a while even after it ends as they are so far away. If it is a closed off area and they can't run, then yes they won't have far to go, but in that case they don't get a save either, they need you to lose concentration or a minute to expire before they can start fighting again (without their weapons).
In play I've seen fear used and it was ok. I've seen HP used to devastating effect by different groups and with multiple DMs.
I have seen exactly the opposite with multiple DMs and at my table. I have seen Fear regularly dramatically change a battle where HP puts a few guys down for a turn or two.
One difference you didn't touch on is the drop effect you get from Fear. With a lot of enemies that means even when they get the opportunity to come back in the fight, they are now unarmed and far away from where they dropped their weapons. If they use natural weapons or if they have a backup on them that might not be a big deal. But if not they are severely nerfed even if they come back.
Maybe different DM styles, but if your DM was letting enemies hit by Fear take extra actions in addition to the dash action then that would explain why fear was not as effective.
