Perhaps you can borrow an idea from my campaign world.
One of the few independent city states that survives on the western coast of the Sea of Swords is the small city of Dee. Dee has neither army nor navy. Dee does not even have a wall around the city, nor a fortress with engines of war overlooking its fine harbor.
Dee has The Master.
In the center of the city atop an unnatural spire of stone stands a fortress of proportions so immence that they stagger the imagination of travellers from Daros or Tether. It is as if some giant hand has stacked uncountable castles atop each other as a child might stack blocks. A vertible mountain of towers, keeps, outbuildings, and battlements on above the other in a combination of chaos and cunning design. No road nor path leads up to this fortress, and none are foolish enough to try to ascend the heights of The Master's Spire.
With two exceptions, The Master of Dee does not tax his subjects, nor regulate them, nor pass over them laws of any sort, nor indeed communicate his wishes with them in any way. If the Master wishes something done, it is simply done. The Master has but two requirements. First, that the privacy and tranquility of his retreat never be interupted. Long ago the armies of Tether marched on Dee. They and all thier baggage disappeared into the dust, never to be seen again. The Master abides no change in Dee that might interfere with his tranquility. Newly constructed buildings are sometimes found taken apart and thier blocks laid in neat stacks if they do not meet with The Master's since of aesthetics. Old buildings are sometimes found fully repaired in the morning. Storms which assail the city are often simply dispelled in a moment. Cool breezes are called in from the Sea during the summer.
The second requirement is servants. Once a year the master sends his Gargoyle servants into the city to steal away those upon which his eye has fallen. They are always young, most are female, and they are frequently handsome of form, but otherwise there are no patterns in who is chosen. Rich or poor, gifted or dull, atheletic or blind, all are exposed. What becomes of those that are tithed is not known to any in the city, though rumors of course abound. Some say that they are taken to an earthly paradise. Others, that they are sacrificed to the dark god that the Master serves. No one knows anything.
The party is part of such a tithe.
Of course, you don't have to have the mystery of the Master's Spire the same as it is in my world (and I'm not going to tell you The Master's secrets anyway), but clearly virtually anything could happen once the party is spirited away by such a person.