Giants rule the world--the giant world. Dragons rule the dragon world, beholders the beholder world, etc.
Just because one species isn't busy keeping another species under their thumb doesn't mean that they rule them. Take our world, for example. It's an insect's world. Of all the species described on the planet, over half are insects. That includes trees, bacteria, worms, birds, us, you name it. By some estimates, over one third of all the biomass that is in terrestrial animals--elephants, rhinoceros, dogs, people, etc.--are in ants and termites. By any objective measure, this is an ant's world, we're just living on it.
Now, of course, this doesn't change the fact that on a day-to-day basis, what we humans do is far more important to us than what the ants do. It also doesn't change the fact that we can wipe out ant colonies far more effectively and with more certainty than an ant colony could kill even one human. (An ant colony, given the right conditions, could kill a human, sure, but it's not a foregone conclusion.)
So, as far as the giant or dragon is concerned, the same thing applies. We are their ants. They probably have built buildings and systems that humanity can only begin to comprehend. Perhaps the very existence of magic is actually something the dragons set in motion. Just as a mouse who finds a bag of grain has hit the jackpot without knowing anything about agriculture, a wizard's spells might be just individual grains to the draconic system of magiculture.
So, why haven't the giants enslaved us? For the same reason we haven't enslaved the ants. It's not worth it to us. Sure, we use ants from time to time--we need them in films, kids have ant farms, etc., but we don't run around forcing them to do our bidding against their will. We'll kill them without remorse if they invade our homes, but they can usually live in our front yards just fine.
But then again, when we do use ants, how aware are they that they are being used by us? How do they know that they're being kept in a small plastic box for the bemusement of children? And, if the giants were to enslave us, how could we tell?