No, they don’t. The DM has narrative control in the fighter case: if the bandits are not already in line of sight (so the fiction was already established), they can have the bandits come before the wall is built, while the wall is being built, or after the wall is built.
In the wizard case, same premise, the wizard has narrative control. If the wizard chooses, there is a permanent stone wall before the bandits arrive.
Similarly, consider an encounter during the night with undead. Without the wizard, the DM has narrative control. The party encounters the undead.
With the wizard, the DM has less narrative control. The undead have no way to bypass Leomund’s Tiny Hut, so the encounter only happens if the wizard chooses (and will probably fight from inside the dome).
The first Interact with an object is a free action, as per the rules. Since ghouls are attacking with their claws, they really don’t have a second object interaction. So the fighter gave up their action to temporarily inconvenience the ghoul until the start of its turn. Yay?