Or if not allied tribes what about allied races. Have them allow Goblins, Gnolls, Giants, Trolls, etc. into the city. While the Orcs themselves are not openly harsh or violent they do not control the actions of those they let in, thus fear and violence from the other races keeping the Humans...