I'm not sure I agree: the only feature you're not using is the second half of Hex Warrior. You still get the benefits of more armor, and Hexblade's Curse, Armor of Hexes and Master of Hexes all work just fine with eldritch blast. Accursed Specter doesn't but it doesn't really work with Pact of the Blade either. Sure, that reduces the weapon proficiency thing to a ribbon, but Pact of the Blade is a ribbon unless you take a bunch of very specific options anyways.Something that never sat right with me was the fact that you could make a Hexblade that didn't have Pact of the Blade, and it was clearly a trap. All of the best weapon invocations are limited to Pact of the Blade, which means you're missing out (especially of a multi-attack feature) if you don't go Pact of the Blade. There are some kewl things you can do with Hexblade and the other two Pact features but really you're missing out if you don't take it.
Heck arguably the most optimized Hexblade is a tomelock.