So, let us consider - Newton went to Trinity College, in Cambridge.
Even today, that college takes a whopping 200 students a year, and only some of them are mathematicians.
In 1650, the population of Europe was about 74 million people.
If European colleges were putting out a thousand mathematicians a year, it would still take 700 years for mathematicians to be one in a thousand people. Ergo, pre-Renaissance, mathematicians were not a one-per-village kind of thing.
The thing you miss is not that mathematicians need to be plentiful for people to learn and extend mathematics - what you need is for them to record mathematics, and pass it along. Mathematics was passed on and was advanced because they communicated through books, and then through colleges, not because you could easily meet a practitioner on the street.