At best, you are effectively eliminating the need for primary ability scores for classes since everyone can use their favorite ability score anyway. 4e did this to a certain degree with the Melee Training feat (which was a patch because many classes used weird stats on Int, Con, or Cha to hit with melee attacks, but were forced to use Str for Basic Attacks like OAs) or 3.0 Psionics, where every discipline was tied to a certain score (Str for Psychmetabolic, Wis for Clairsentience, Cha for Telepathy, etc). Note that both of these were not especially well received (as 3.5 psionics abandoned it, and 4e viewed it as a feat tax).
At worst, you are going make every class MAD, esp if some of these abilities don't look especially racial (or put another way, there is no reason a wizard with an 19 dex can't move after passing a save, but an elf with a 14 dex can). I guess you can say you have to specialize based on your ability scores (so you can't have the dexterous wizard bonus, the strong wizard bonus, or the tough wizard bonus, you have to choose) but that still ends up limiting certain options to certain races (why should goliaths be better abjurers, or gnomes be better battlemasters?) but you've broken it down the suboptional stage rather than the class stage.
Personally, I think at the point you'd be better untying ability scores from class altogether, much like how it was in OD&D. You could have spell attacks/DCs purely based on class level rather than proficiency+ability mod, for example. You could also institute some manner of weapon specialization to allow warrior types to rely less on raw ability and more on training and class-derived bonuses. You could also slow down the bonuses from ability scores to something closer to Basic D&Ds (max +3) so that while a strong fighter might have an edge in raw talent, a dedicated gnome fighter could close the gap with class ability.