I'm of the opinion.. why not? Have huge numbers of groups. Even if you have 1% of the population as 'heroes' then in a 500,000 person nation, that still means 5000 people. Some few should be rich and retired (and anyone rich enough to retire should generally be famous, since the profession tends to have a high mortality rate) but most should be struggling. That doesn't necessarily mean that they're all level 1 weaklings.. just that they haven't hit the 'big score' that the retired ones have. So most of them should be mercenaries, thieves, assassins, and others in unsavory work, or have left that job. Others should guardsmen, of course, but they should be the higher-ranked ones that still patrol or be the 'chief who took a desk job after an illustrious career' type. Others are probably nobles, and only adventured for a little while to gain enough fame to make themselves more notable at court.
Mostly though, they should be described as "Those stupid idiots who went off and left the village... and will probably never return." (Bonus points if that's what the villagers of the PCs' hometowns think of the PCs.)