Part of it I think is how we have become conditioned to look at characters. We almost always come at it Class-first. Meaning that whenever someone asks "What are you?" It's always "I'm a Fighter." Or "I'm a Cleric." Never "I'm an herbalist trying to catalogue all the plants in the kingdom." Or "I'm a knight in service to my Lord." We don't use our race, or our background, or our bond, or any other description of who we are or what is most important to us or what we are adventuring for. It's always "I'm a Paladin." "I'm a Monk."
As a result... we end up with how you are seeing things, which is "All Sorcerers are charismatic!" Or "All Druids are wise!" When in point of fact... it's never been the fact that you are a Druid that makes you wise... it's the the fact that because you are wise, you become a druid.
Why aren't there many charismatic Wizards? Because you needed to be extremely intelligent to be able to become a Wizard. Your high intelligence is what drove you to the wizardly arts. So while you might also have a mid to high charisma... it's your intellect that made you become a Wizard. But if you grew up focusing on your charisma, you probably were more likely to use your personality to become a Bard. Or you were more likely to be able to use your personal presence and authority to stand up to a patron to make a pact with them and become a Warlock.
That being said... there's also no reason why you couldn't change the main ability score for your spellcasters. This is especially true of the Cleric and the Wizard, as they are influenced from the beginning from their Domain or their School... and both of those can make sense to use different ability scores as their primary stat. I mean just off the top of my head...
Domains:
Arcana - INT
Forge - INT
Grave - WIS
Knowledge - INT
Life - WIS
Light - CHA
Nature - WIS
Tempest - CHA
Trickery - INT
War - CHA
Schools:
Abjuration - WIS
Conjuration - CHA
Divination - INT
Enchantment - CHA
Evocation - INT
Illusion - INT
Necromancy - WIS
Transmutation - CHA
If you decide you want to vary things up with the ability score to the class... just use a listing like this. I don't think you'd want to do it with all the classes, as many of the classes/subclasses don't lend themselves necessarily to differing ability scores. I mean, all bards seemingly should use charisma as their stat, and the the different types of Lands of a Land druid do not imply any reason why they would be INT or CHA instead of WIS (Oh, you're from the coast? Then of course you're charismatic!)
But the wizard schools and the cleric domains absolutely can imply which of the three mental ability scores could/should be more important. Knowledge domain is INT! Enchantment school is CHA! Many of them are fairly obvious (and you can then assign the others to keep them thematic and relatively equal in number, like the life & death domains/school assigned to WIS.)
I'm actually thinking about doing something like this in my next campaign, since I do like the idea of using only the four Basic classes. And this would give me a way to do so while not turning CHA into the dump stat it used to be, and to vary up the types of Clerics and Wizards we would see.
But the biggest thing is to just stop thinking "Class first" for whom your character is. No longer "Oh, I'm a Monk? Then of course I'm Dexterous and Wise!" But rather... "I grew up extremely wise for my age and was always agile, and thus I joined a monastery when I became of age to focus my mind and body on enlightenment."
