The problem here is the exact same thing can be said about many other societies in-game as well. If you play a game where there's evil monsters (literal or humanoid) terrorizing the countryside, or where the political or religious ruler is corrupt and doing evil things, then by this definition any villager (human, elf, halfling, whatever) who knows about it but is just trying to get by, unwilling (or unable) to stand up to them, is at least a bit evil as well.
Heck, how many games have featured horrible monsters or evil armies attacking one area when, just a hex or two away according to the map, there's a city of good aligned elves or dwarfs doing nothing until the PCs think to go to them and get help (which itself usually involves the PCs having to convince them to join in).
I don't know why the typical drow commoner is considered evil, even lowercase-e evil, when the typical non-drow commoner who does the exact same thing isn't.