That's fine, but until level 11 when fighter gets 3 attacks, they really aren't much weaker. I mean, i know a lot also depend on subclass selection, but overall I don't see either the paladin or ranger as much weaker in fighting that the fighter--even without their spells.
We are talking about a world and system where rangers, paladins, eldritch Knight, duskblades, and swordmages are subclasses of fighter with champions and battle master.
It kind of breaks the immersion and suspension of belief if a fighter can eschew magic for more focus on martial combat and not be much better than than those who do
Why would a fighter not learn magical if those who do learn magic lose almost nothing in return?