Remathilis
Legend
The issue is that spell points are strictly better than any caster in the game.They could also solve it by keeping Pact Magic but using the optional spell point system from the DMG. That way, a warlock has enough points for two max level spells, or maybe one max level and a few low level if they choose. Keeps the separate flavour, but still gives the warlock a little flexibility.
A 7th level caster has 38 spell points. A 4th level spell costs 6. Meaning they can cast six fourth level spells with (6x6) plus have two left over for mage armor and a shield. That's on par with a 7th level warlock going nova and resting twice, or better than a sorcerer can do with Font of Magic.