I don't mean that the duergar would be afraid of others - but it is a city and it does have gates, like any city, you can't just walk in. They might need a very good reason to gain entry.
And, the party might not look like the kind that would normally have business in the city (i.e. Drow...