Isn't magic item distribution down to the DM mostly? 5e is designed so PCs don't need magical weapons, so having to do without them is hardly "punishment".
It's up to the DM in a hypothetical situation, and designing the game from the assumption of no magic items was a good choice, but I would definitely go so far as to say that this is not the intent of the game. Both for old fans and new ones, the allure of getting new powerful gear is a strong one, whether it be random or placed. To my knowledge every AP released so far has had a decent number of magic items in them, some of which are very powerful.
Likewise, while losing one doesn't cripple the character as in prior editions, being parted with a favored and storied magic item you may not be getting back is punishment enough, yeah.
I think if bandits had the cash and opportunities to purchase any weapon that they wanted, they might not have picked such a hazardous lifestyle.
There are a lot of causes to turn to banditry, and not all of them leave you poorly equipped. Even if they did, that would just be cause for a small blurb about dealing a smaller amount of damage in the stat block, same as bugbears and some of the orcs in Volo's get arbitrary extra damage dice. That option is waaay more attractive to me then trying to justify a bad weapons table.
How is the spear a "pretty bad weapon"? It does only 1 point of damage less than the Longsword in exchange for being throwable. Even if you aren't using the feat that specifically supports it, that isn't a bad exchange.
It's bad for several reasons:
1) Multi-attacks are the go-to damage amp for most martial classes, and so each point of difference is significant. Realistically by midgame this can cost you 2-5 damage per round based on various circumstances. Whether or not that matters you is your call I suppose
2) The ranged quality is negligible on spears. At 20ft or closer chances are you just move to the enemy and attack, and further than that it gets disadvantage with poor damage. In fact, unless you're using the versatile property, you're just straight-up better off using a javelin at all times, since their 1h damage is the same and it has marginally better range.
3) Enchantments. This certainly does fall prey to the "no assumed magic items" argument, but you can flip through the DMG and see how many named/specialized swords there are versus spear options, and swords are definitely the victor.
This is what I mean when I ask for weapons to have a purpose/niche to fill that makes them an attractive choice in some way. Unless I'm dead set on using a spear for RP reasons, I can't think of any class where it would be desirable to do so.