Agreed, which is why there needs to be a lot more "meat" to those first few levels.
What you point out here is nothing more than a design flaw in 5e that, while getting play to the "sweet spot" faster, really does mess with believable worldbuilding in the process. It's kind of a carry-over from 4e which (in a different form of poor-for-worldbuilding design) had its PCs start with nearly-3rd-level abilities right from the hop and left a huge gap between commoner and 1st-level PC.
I mean, I've no problem with there being lots of 0th and 1st and even 2nd level types in the setting provided the progression is reasonably consistent - soldiers, street thieves, etc. who have built up over some years a degree of experience that adventurers tend to get in a few weeks - and my take is that anyone in the setting can earn xp if they try to. It's just that most either don't try (a typical baker or farmer), or don't realize they're very slowly earning xp as they go along (cloistered cleric, street thief, etc.).