So Elves and Bards are like the Dark Side of the Force. It's the power that tempting; never mind the evil. No matter how much you try to tamp them down, they will still be there. Doesn't give you license to succumb and murder all the younglings.
But I am relatively unworried about the idea of people making spell casters that are competent in melee without spells. That's the idea.
The issue of half-casting melee classes (Paladin, Ranger) is something I haven't really thought about yet- I think many people enjoy the various ranger variants that do not have spells for a reason. As for Paladins, the issue is primarily smite; honestly, other than not allowing smite but continuing to allow them to use spells for other reasons, I can't think of a better proposal. But that's why we have the wisdom of crowds, right?