Slavery isn't in and of itself evil.
However, the capacity to do evil, particularly in the context of slavery (master/slave relations, slave acquisition, etc) is great. It's one of those hazy areas, like negative energy, where the subject itself isn't inherrantly evil, but evil can be done -with- it.
Slavery can mean 30 guys sleeping in a bare room, barely fed enough to stay alive, worked nearly into the grave, and horsewhipped at the slightest prompting. Rampant dehumanizing abuse, and easily identifiable as evil.
Slavery can also mean having legally protected status - noone messing with you because if they do, they're not just messing with one guy, they're messing with the king/church/etc - benefiting from an education far above that recieved by the common man, having an important and assured position in the upper eschelons of society, and a quality of living well above the norm. Being a valued member of society, provided for and respected. Is this evil?
In some societies, slaves were treated as worth less than farm animals, and the majority of their owners tried to claim some kind of inherrant superiority over their slaves. We're better than they are, they're meant to be slaves, and it's our right to own them... uh... just because.
Other societies treated slaves with care and difference because a slave was A) a highly useful, versatile, adaptable commodity, and B) expensive as all hell. Mistreating them would be like buying $250,000 car after $250,000 car and just driving them off a cliff, one by one. Or buying a super computer and then gutting it and using it as a fish tank.
Ultimatly it's no more inherrantly evil than any of a number of things. Like swords. Or taxes. Or magic. Or fire. It's all in how you use it.