Sure, you can invest only a small amount into Eldritch Blasting if you want to. In that case, I don’t see how this interferes with your ability to make a witch. Is it not a witch if you have an effective attack spell?
When I said that a lot of the warlock's power is tied up in EB/AB, what I meant was that the class allocates a huge amount of its "power space" to that combination. The warlock chassis is built around a powerful at-will attack, a couple of spell slots, and a smattering of minor tricks.
You can replace the powerful at-will attack with a weaker one, but all you get in exchange is one more minor trick (the invocation that you didn't devote to Agonizing Blast). You can play a bladelock and have a
different powerful at-will attack. But no matter how you slice it, you're playing a class whose primary function is smacking people for damage. All other warlock features are competing for the leftovers. Even
hex, the warlock's signature "curse" spell, is mostly just a way for you personally to deal more damage. The skill debuff is occasionally useful, but as curses go, it's no great shakes.
And that's not a bad thing! Plenty of classes smack for damage as their main function. It's a perfectly fine way to design a class. But I don't think it's a good way to design a
witch.
When I think of a witch, I imagine brewing potions that weaken your enemies and strengthen your allies. I imagine turning yourself and other people into animals, and calling upon animals and plants to aid you. I imagine laying curses to confound your enemies, bewildering them with illusions, or outright controlling their minds. What I
don't imagine is "smack for damage," and that is why I see the warlock as a poor fit: Too much of the class is devoted to doing a non-witchy thing.
(Although in the course of writing this post, I have more than half convinced myself that the best chassis is bard rather than druid.)