I wouldn't say
completely wrong, but you're right that it misses out a key part of the situation:
At low levels, the total modifier to any dice roll that a really specialised character can muster is fairly low, and not hugely different from the total modifier that a non-skilled character can muster. So, a 1st level Fighter might have +3 or +4 to his attack rolls, whereas the Wizard might have +0 or +1. When compared with the d20 roll, this difference is fairly insignificant, rendering character roles very limited.
On the other hand, at high levels, the difference in modifiers is huge - a 20th level Fighter might easily have +40 or more with his primary weapon, while the Wizard has +12 or so. This means that any AC that the Wizard can hit
at all is trivial for the Fighter, while any AC that is challenging for the Fighter is completely out of reach of the Wizard. This in itself wouldn't be a problem, except it applies to all areas of the game - attack rolls, damage output, saving throws, and so on and so forth.
On the other hand, in the middle range, the differences in the total modifiers are large enough to be significant, but not overwhelming. So, an AC that the Fighter might hit on a 5 would require a 15 from the Wizard - it's much less likely, but still doable.
Does that meet with your approval? Or is it still nonsense?