Agreed. I think the simplest fix to Eldritch Blast spam is to remove all of the Eldritch Blast invocations and then add more invocations to support different types of cantrips. Like change Agonizing Blast to a 3rd level invocation (to discourage dips) and make it do this:
"Once per turn, when you do damage with a cantrip, you may add your Charisma modifier to the damage roll. At 5th, 11th, and 17th level, you may add an additional 1d8 to the damage roll. (1d8+Cha at 5th level, 2d8+Cha at 11th level, 3d8+Cha at 17th level)"
That keeps the Eldritch Blast damage scaling in place (Agonizing Blast adds 20 damage at 17th level, this is 18.5 but can also crit), but makes other cantrips do roughly similar damage as EB spam.