There's almost always multiple casters in even podunk villages. Compare that to today where there's often not a GP, let alone a specialist like an OBGYN in areas.The published adventures are not representative of the world as a whole. They cover exceptional events more or less by definition, and focus on the activities of exceptional people.
D&D wants to pretend it's low magic to let people imagine it's Tolkein. The reality of the setting and published material doesn't reflect that. Sort of like how in 1st edition magic items were "rare" then you'd pull double digit permanent items out of a module, and if you actually rolled per the treasure tables it was clear that magic items were not.