So effectively, abstract that further to a party wide skill challenge.
The familiar might be one way of acquiring knowledge for the wizard PC, but the bard might go and busk across the street, the fighter might bribe an off duty guard, the rogue might spy for the rooftops for a night.
I do...