What makes you say that? Are you trying to use the familiar in combat?
How? Both as in "what does it do?" and "how does it survive?"
Our warlock has an Imp that's used to scout ahead, deliver messages and be, well, impish.
As a flying invisible superior darkvision scout it's almost without peer. And oh, it's unkillable too (in that it can be resummoned if it dies)
But in combat it generally is tucked safely inside it's master's vest.
While lots of people expect an animal companion to be present in combat that's not the case for familiars. And thus I don't see a direct comparison between the two as regards action economy - most often when the familiar acts, the master doesn't act, because she's not even there...
Presumably you don't claim there's an action economy issue comparable to the poor poor beastmaster...
I don't think I've ever seen 5e characters retreat for the day unless they were pretty spent with no way of recovering meaningly during a short rest.
Now, if you want to claim that the 6-8 encounters per day takes some work to pull off, you'll find no disagreement.
Everything else a warlock gains has to take into account how crazy good their at-will (and almost at-will) damage output is compared to every other caster in the game.
That's exactly what happens though. The party retreats when their resources are expended. They may have burned through these too quickly, and in theory should be left plinking away with cantrips for the remainder of the day's 6-8 battles. But unless there's a time constraint the majority votes to fall back. If the short-resters were to push them onward knowing they're spent, they'd be risking character death or possibly TPK. There's less punishment for those who overexert themselves, but is risk to go onward when your teammates can't.
But wouldn't that still be broken though? EB is broken by being so obscenely good, that the remaining 17-18 levels (the majority of the class) therefore has to be fairly weak to balance. Overall the Warlock 20 can do reliable damage, but the "fun factor" isn't there because it all comes from one singular ability.
That's what needs to be redesigned - more fun, more options, less constant EB spam and more incentive to go further than three levels.
I just wish some people would bother actually playing certain things before going around making bold assertions as to perceived inadequacies...
I love my level 10 warlock, I seriously can't wait to hit level 11. I play a Tiefling tome pact with a fiend patron. My DM let my Patron teach me Wall of Stone as the only patron option 5th level spell. I'm in a kingdom building style of campaign so that helps a bit outside combat.
I've got a Grimoire that I found that is all mysterious and I know its sentient, but it won't talk to me yet. I know it hungers for worthy souls who have to die by my efforts though. Whenever I kill something that's "worthy" only I can see a incorporeal greenish soul get sucked into my Grimoire. There's more power to be gotten from this book but I am at a loss for now on how to unlock those secrets. It also acts as my Book of Ancient secrets as well.
All in all I love the class but at times I do wish I had more options or WoTC would add in more spells & invocations. I also would have liked to see Hunger of Hadar as a spell that scaled a bit more with my levels. I used that a lot when I first learned it.
Also I used Hex a total of 3 times in those 10 levels. I will get more mileage out of it when I hit level 11 though.

(Dungeons & Dragons)
Rulebook featuring "high magic" options, including a host of new spells.