For attacks of any type including spell attacks, the "standard" character optimization number I see is a 65% chance to hit assuming your prime ability score is raised when you can. So mid tier, where it's expecting a 20, a 16 would be a 55% chance to hit. That's about 15% conversion of hits to misses with the lower score.
But that's the most affected number. Saves are another deal. Because of the nature of them, a monster will likely have at most four good saves between high modifiers and proficiency, so it will have two or more saves that do not really scale with their CR. A character having a spell DC that isn't scaling either is even, and with a 16 is still pretty good. So if you have the flexibility to pick a wide selection of spells against different saves, your ability score modifier isn't a big deal. On the other had, a class like the bard with primarily WIS saves could find themselves in trouble with only a 16.
A few casters get additional effect from their ability score. Healing adds the modifier in - a Healing Word for d4+3 (avg 5.5) vs. d4+5 (avg 7.5) loses more than a quarter of it's effectiveness. A warlock's EB with Agonizing Blast adds +CHR to damage, on top of it already being a spell attack, so is very sensitive to that lower score.
Many classes/subclasses also offer modifiers on the casting ability, most commonly uses per rest but sometimes a DC or effect modifier. The bard gets a decent amount of power out of one of the few non-Concentration buffs that also has nifty usage by subclass with Bardic Inspiration, and 3 per short rest is 40% less than 5 per short rest.
How often skills associated with your prime come up in play is also a factor. A CHR caster who also is the party face will find that 16 vs. 20 comes up not just in more rolls, but in more rolls that aren't averaged away by the large number of rolls in combat. In other words non-combat pillars are often resolved with a lot less rolls so each one is more important to succeed, even though the d20 is very swingy. WIS casters have a selection of useful practical skills including the ever-popular Perception, while INT casters have a selection that is often not replicated by other characters with the possible exceptions of Religion or Investigation, and even there has a big advantage.