Yea but it is. The different HD of fighter / barbarian vs fullcaster make a difference. Also you do not give the ASI which are essential for e.g. fighter to get max to hit and max damage even if not using a magic weapon.And I have already given some non-casters extra features as well. Features in this first draft may or not make it into the final cut. I don't see this really as an issue, either.
Whereas for a mage the less than 20 only increases the chance for a save slightly.
It would be easier to leave the rules like they are tbh. and only run campaigns up to 10, or to make it mandatory that for some ASIs the player has to take a feat instead if ASI is the thing you got beef with.