Rhenny
Adventurer
Isn't the entire point of passive perception and similar things the fact that we now don't have to have this situation? Only one person is doing the rolling, rolling against their opponent's static target. All having both players roll does is alter the variance of the contested situation. If we wanted to do that, we could make all rolls 2d20 rolls (or 3d6, or any number of things that are equally arbitrary).
That's exactly how I use passive checks when players/PCs shouldn't know how easy or difficult something is. I roll a d20 as DM and add modifiers that I think might be important (using the monsters active perception, or other skill that is applicable, or I just make it up).
I also roll a die when I'm not sure if the task is easy, moderate or difficult, or if I don't feel like setting the number before hand.
It works fine to me.