An evil AND destructive society strikes me more as Chaotic Evil, anyway. A Lawful Evil one is all about maintaining society and the status quo for the benefit of those at the top. By my reckoning, at least.
Rome, being the expansionist, slave-using, manipulative nation it was, whose rulers tended towards murderous treachery (even discounting figures like Caligula), was, as I see it, an essentially Lawful Evil society. For that matter, I'd place most societies throughout history and the modern era as being some form of Lawful or Evil.
Now, that doesn't mean the people are, just the government, its policies, why those policies are set up, and how they're implemented. Even an act which is essentially Evil can benefit others - it's just that it benefits whomever performed it the most, and the benefit to others was either a side-effect, a way to placate them, or to create the circumstances which led to the evil individuals greater fortunes. I'd say that's often the case with government and business.
With that in mind, one can have an Evil government presiding over an essentially Neutral people. I'd say that's normally the case in most Evil societies (or Good ones, for that matter).
Evil doesn't mean destructive, at least, not necessarily. Most often it just means selfish, in a way defined by the ethical half of the alignment. Lawful Evil bends rules to it benefit, Neutral Evil uses any means necessary to get what it wants, and Chaotic Evil acts on impulse in a selfish manner. To boil things down, and by my own generalized definition of alignments.
Therefore, a city based on slavery has strong inclinations towards evil. Perhaps it only enslaves criminals, which alleviates that to a degree. Then again, killing people for entertainment (gladatorial matches) corrups the act once more. More laws governing the treatment of slaves generally doesn't change the Evil, moral aspect of it, only solidifies its ethically Lawful position.
In regards to a Chaotic Neutral society and slavery, it doesn't strike me as the sort of thing being particularly common. Slaves might exist, to be sure, but I don't believe there'd be any laws, or individuals who would enforce any laws on slavery that did exist, in such a city. Chaotic Neutral has a general "I'll do what I well please" kind of attitude going for it, without the addition of the Evil "...and what I please is to force other people to do as I please as well." So it would start spiralling towards something more Evil.
As said, though, there are a few circumstances that might elevate the act to neutral, such as only making slaves of criminals, but, yeah, it has a decent root in evil, no matter how you cut it, and about an equal root in lawfulness, as well (as slavery insinuates hierarchy, as previously stated). And a city like Rome, in my opinion, had a fairly morally corrupt government, with laws, foreign policies and entertainment that supported that idea. Yet, Rome could still be a decent place to live or visit. Evil often manifests itself quite subtly, after all.
Anyway. Bit simplified, perhaps, but, for now, I'm done.