The party should be getting a stipend from Brevoy to found your kingdom there. By rights, and by charter, this land is yours.
I'd try diplomatically to annex the place, and kick them out by force if they were decidedly hostile.
However, to be frank, it sounds like the DM is playing the...