So if they don't take the class they aren't magical, well... what if they multi-class? What if they take a feat? Also, what about wizards?
See, wizards are actually a huge problem for this view. Because a wizard can be created (and we have canonical, official proof of this in Xanathars) by stealing a spellbook and reading it. Or by finding a spellbook in an old ruin and reading it. No training, no magical infusion of power, just picked up a book and started self-teaching magic. Now, you can shrug and say "well, clearly they had magical potential and just got lucky" but... we know that EVERY PC therefore has magical potential. All of them. You cannot create a PC to whom magic is not an option during your career. Sure, you can declare "my PC is completely non-magical and will take no magical abilities!" but if you change your mind... nothing mechanically about your PC changes.
So, why isn't everyone innately capable of magic? The only possible reason to say so is because not everyone learns how to use their magic, but by that logic you can declare that not everyone is innately capable of being a car mechanic, because not everyone learns how to fix and care for a car. But that isn't an issue with lack of innate ability, that is an issue with a lack of KNOWLEDGE.