I like the idea since it allows for more diverse character concept (the charismatic barbarian, the strong monk, the intelligent fighter, ...) without making you feel like you're not making full use of your class abilities. It also mostly solves the problem of racial stat bonuses pushing players to certain race/class combos, in a better way (IMO) than just allowing moving the stat bonuses.
So the Barbarian who focuses on Charisma does as much damage as the Barbarian who focuses on being strong (which usually has a visualization of being more damaging)? Same as fighter with Intelligence vs. fighter with strength? Why aren't the exceptionally strong barbarian and fighter worth keeeping?