FrogReaver
The most respectful and polite poster ever
In an ideal world (which none of us ever have!) the dungeon or complex or adventure is fully designed before its first contact with the players/PCs.
When hit with a curveball you've no choice but make something up (hence my saying we never have an ideal world!) and if they latch onto it as a red herring that's fine.
The harm doesn't come from the specific moving of whatever element(s), the harm comes from and through the motivation for doing so.
If you're trying to make things easier for the PCs e.g. putting a clue in their path they'd have otherwise missed, that's a harmful motivation. You've broken your neutrality as DM.
If you're trying to make things harder on the PCs e.g. moving some enemies into a nearby room as the PCs have in your view had it too easy so far, that's a harmful motivation. You've broken your neutrality as DM.
And it'd be extremely rare that a mid-adventure element change would be done entirely free of either of these motivations. Mistake correction (e.g. you've suddenly realized some maps of areas yet unexplored don't line up as they should) might be one.
I think that unless one has the whole campaign planned from day 1 then there's no way to be truly impartial as a DM. Consider what happens when you are planning your next sessions dungeon. Are you taking into account the PC's and their abilities when designing the dungeon - if so then you aren't actually impartial.
What you are actually trying to do is set up a dungeon that will challenge the players you have - ie being partial and making decisions based on the party you are DM'ing for.
Now fast forward to mid-session. You PC's are struggling to get through the dungeon you thought would be a cakewalk for them. You are the one that engineered the dungeon taking into account their abilities but apparently you failed at designing this dungeon to your specifications. IMO at this point it's your failing and not theirs and the whole campaign would be best served for you to take action due to your improperly engineered dungeon.