While things have changed, in Ad&d, you rolled your ability scores (Method 1) 3d6, in order. A Wizard needed a minimum of 9 to be a wizard.
Ok, certainly not a huge bar, but you had to roll that 9 or higher in your 'fourth' 3d6 roll to qualify.
However, with that 9, you could only cast up to 4th level spells, only had a 35% chance to learn a spell (re-rollable each level gained), and a maximum of 6 spells per level (that you can know, granted, this last is optional).
So, yeah, you could be "average", and learn to cast spells, but not many, and not to a high level, and you were likely not learning spells very easily as you came across them.
If I was rolling up a character, and didn't roll well on that 3d6 for Int, or it wasn't my highest stat, I wouldn't be making a Wizard. With d4 HP, and a high xp chart, it was a tough road to hoe to get to that "powerful wizard" mantle.
In the worlds I run, I base my "level of magic" against those types of parameters, so magic is "rare" and looked at askance or feared, there are no wizard colleges, so you learn either in a major city combing through whatever stands for books and tomes, or you have a mentor, and you don't pick your spells, they're randomly generated early, and you have to find (or do tasks for your mentor) to gain more if you're adventuring.
Its not an easy path.