Now that I think about it, spell DC is 8+Prof+ability mod, while passive checks are "10 + all the modifiers that normally apply to the check", so it sems even worse than what you say.
Unless there's some reason to apply disadvantage to the check (synergy with some fog/mist/darkness effect), it...