Yes, I think you nailed it.
Ultimately, preventing the stacking makes life much easier for the GM. If he/she wants to impose a Disadvantage, because it would make for a more dramatic and exciting situation, then just needs to come up with one source for it, rather than several to counter all the...