There are multiple justifications, but with regard to the context of presenting it from a standpoint of world-building, the justification is verisimilitude. Simply put, there's a point where all of the underlying factors that should result in institutionalized slavery are present, and so its not being there becomes noticeable enough that it impinges on suspension of disbelief.
For most fantasy role-playing games, the presence of a medieval (European) culture is presented as a pastiche rather than any sort of serious historical work. That said, part of that pastiche involves certain aspects of the cultural development of the societies involves, which includes a marked lack of civil infrastructure and social support networks. The result is that it's taken as self-evident that such societies necessarily involve their people living under a greater degree of threat, not only from monsters, criminal elements, and other mayhem, but also from what we'd describe (for the sake of convenience) as ill-fortune, such as sicknesses, poverty, famine, etc.
That's important to take into account, because even in settings with semi-ubiquitous magic, interventionist deities, and all sorts of lifeforms, it's virtually always understood that none of these undercut the aforementioned medieval pastiche in any meaningful way. Specifically, scarcity is still an economic reality in the context of the game world, as is natural selection and other forms of Darwinism.
The result is that all of the reasons for enacting slavery are present, and most of the controls which would be able to prevent slavery's institutionalization are either lacking or anemic. A pseudo-medieval world, where survival is still a very real struggle for most of the background characters who make up the setting, is likely either not going to have organizations dedicated to protecting human (or rather, sapient) rights, or if such groups do exist, they're simply not going to have the level of outreach and authority necessary to protect those rights on any sort of pervasive scale.
Meanwhile, there's going to be villainous groups who do have the means, motive, and opportunity to enact slavery, since a workforce which you don't need to pay is a self-evident economic boon, even if it's morally repugnant, and that boon helps immunize them from the aforementioned threats that people in that society live under. The possibility of a slave uprising is likewise fairly easy to control if you keep the slaves oppressed (which goes with them being slaves), and is likewise not going to be a risk that outweighs the benefit(s) of having an oppressed class. And that's without issues of how it obviates the need for jails if you use slavery as a punishment for (alleged) criminal activity, or engaging in genocide if you conquer an enemy in a war, etc. Slavery is morally abhorrent, but if evil people don't care about the moral aspect, it has a lot of pragmatic dimensions to it.
So assuming the setting isn't a utopia, slavery's absence can be fairly stark.
And that's not even getting into the fantastical aspects of it. If we assume that evil deities have a vested interest in people doing evil things, then at least some of them should have slavery as an aspect of their religious portfolio. Devils and other corruptive influences will also push that idea. Magic makes such things possible as well, once you start looking into perpetual charm magic, etc.
There's no reason why slavery has to be present in a setting. But how you present the setting necessarily opens it up to questions about its internal logic and self-consistency, which means that if you have all of the underlying ingredients there, it becomes odd that nowhere has anyone put two and two together. If we take it as a given that people want to interrogate how their settings work, because a greater understanding of them is more enjoyable, then laying all of the framework for slavery, even inadvertently, and then having it not present is something that people who enjoy world-building are going to notice and comment on.
