This suggests any improvement to those attributes must be one those casters are incentivized not to use. E.g. for Charisma it might be something like:Buffing int,Wis,cha is just an implicit caster buff, because they the only ones main stating them outside a few niche builds that can attack with cha or such.
As long as your Charisma modifier is at least +1, you can communicate to an ally to give them Inspiration. Doing so reduces your Charisma modifier by 1. It returns to normal when you complete a long rest.
I'm envisioning characters yelling encouragement at each other. Since it would be reducing an attribute modifier, it should discourage Charisma-powered classes from getting a double benefit from their primary attribute.