It’s WAY MORE noticeable on casters because in addition to missing more often and having their spells saved against more often, they can prepare fewer spells.
Maybe you haven’t seen this happen, but I absolutely have. Missing 5% more often is noticeable, especially over the course of an adventuring day, especially in a bounded accuracy system like 5e’s. When players, especially new players, notice that they’re doing the same stuff as everyone else, but consistently doing just a bit worse at it, they get frustrated.