Still doesn't stop the "supervisor" (i.e.: PC) from asking stupid questions, and the builder(s) from sighing and explaining in great detail to the PC why they don't want to have a door/window/wall in that particular place, and perhaps working out some sort of solution that meets the PCs needs without compromising the structural integrity of the fortress. All that time takes away productive "working" time.
And how exactly would that make the work go faster? It would actually slow it down.