Well, Taking 10 was a thing in 3.5 and most people seemed okay with it. I'm fine with it, less chance of RNG causing a relatively highly skilled character from missing an average DC. I think it makes even more sense in 5E where bounded accuracy generally caps a lot of checks to +11 or so for max stat, 20th level characters unaided by magic or Expertise. With that said, I'm not sure if it is appropriate for an opposed test, at least for the "offensive" side of the opposed check. Specifically with stealth, unless the one perceiving is actively maintaining guard, I usually just use their Passive Perception as a DC, modified as appropriate. So, in the case where they may have disadvantage due to loud noises or other distractions, it would be Passive Perception -5.
I think bonuses/penalties should be applied to the test they actually impact. For instance, loud noises don't necessarily make it easier for one to walk more quietly, but they do impact the ability of an opponent to hear, so penalty to the Perception check. Walking around in soft slippers instead of regular shoes, however, make it more easy to move silently, so bonus to the Stealth check. At least, that's how I rule it.