D&D assumes over 50% due to with fighter rogue cleric wizard.If you can have 2 or 3 people in a group with access to magic, that's not low frequency. With low frequency, even 1 PC being a caster would be rare.
That's where Cantrips counting as prepared slots come in. You quickly run out of preparation slots.Having only one(maybe two) school would be low versatility. Banning a single school is just a blip on the versatility loss index
But maybe 3 banned slots would create those ultra narrow casters of some fantasy
You won't present as casters due to your go to being weapon attacks.How is magic not visible if you multiclass? You can still toss around spells/cantrips.