What if the PCs have made a name for themselves knocking over wizards' towers?Of course they can.
All that's needed is to, as far as is practical, design the challenges well ahead of time - as in, before you even know which characters will be facing said challenges.
Then, if-when some characters do encounter said challenges, present them fairly and without modification.
An example: somewhere in the setting there might be a stereotypical Evil Wizard's Tower containing - wait for it - an Evil Wizard. You-as DM know this Evil Wizard is of X level and has Y spells and Z items at his disposal, and that his tower has defenses A, B, and C and also contains other occupants J, K, and L; you know all this because you've made notes and set it up such ahead of time, maybe even as far back as when you were designing the setting before play began.
So now you've got a tower. A neutral-arbiter GM then goes on to present the challenges of that tower as written without regard for the capabilities etc. of whatever characters have decided to tackle said tower, and lets things fall out as they will even if it means the PCs become smears on the pavement or - flip side - the tower turns out to be a pushover for the high-powered group.
I don't think you can set a hard rule that the world must be agnostic toward the PCs in order to be "fair" or "neutral." Of course, I don't think most GMs are neutral. Nor do I think that is desirable. I think the GM should be on the side of the PCs.
But that's different than being fair. And you can still be adversarial and be fair.