Passive perception or active perception? If we are using logic, passive perception should have advantage in combat as your senses are heightened. You are literally fighting for your life and your senses are heightened.
You should have disadvantage on passive perception when you are in the 7th hour of your watch at the guard post and dreaming about what you are going to do with your girlfriend when you get off in an hour.
If it was active perception I could understand your logic, it is harder to actively search while blocking, dodging etc.