I take a different approach.
The feature (position of privilege) allows exactly what it says - basically anywhere you happen to be. Stuck in the City of Brass (elemental plane of fire) - you can get an audience with a noble if you really want to. This is a great way to drive the game forward! And it can really lead to fun and interesting things happening.
To me there has to be a logical reason for the background feature to matter. A lord in the City of Brass isn't going to care that my PC from a plane of existence they know nothing about is not going to have any influence. It's a throw-away line at the end of a long description. The description explicitly says to work with the DM to determine how much authority you have.
Backgrounds are not hard rules in my game, the social aspects of the world still need to make sense in context.