I mean, at the very least passive perception sets a DC for stealthing. It makes stealthing so simple, roll for stealth. OK its higher than these guys, only this one guy can notice you.
The other option is every time someone attempts to hide you get to have every creature roll perception really quick. OR hiding just flipping works, until someone actively looks for them using an action.
Passive perception just makes it all go so much smoother and quicker and more logically.
As soon as you realize that it doesn't see through walls,carpets, furniture, you realize many traps or important objects might still be hidden fairly easily.
"Removing the rug reveals a tile pattern on the floor, a Wisdom (Perception) check of 14 reveals one of the tiles is more worn"
"In the back of the closet, behind piles of clothes is a loose wall panel that can be noticed with a Wisdom (Perception) check of 15"
Neither of these are revealed by a person with passive perception of 20 entering the room. Both require doing something to actually have a chance to look at the hidden thing.
A player who uses a feat to get observant has given up a feat in order to get a benefit of spotting hidden things more easily. He could have just as easily picked Magic Initiate : Druid, and then grabbed guidance, produce flame, and goodberry, now that player is 1d4 better at every skill check that isn't super rushed, can also make a fire, and will never starve to death. He could have picked Inspiring leader and given out tons of temporary hit points, saving lots of player's (and npc's, they don't expire) lives.
But for some reason when people can notice traps or hiding creatures more consistently it just explodes some DMs minds. I am curious if they are just as quick to raise the DC of every trap in the world if the player picks expertise in perception.
Randomly rolling a perception check from the whole party every 5 feet isn't my idea of fun. If you want a trap to be an important part of a dungeon, make it an important part of the dungeon, not a roll x or take 3d6 poison damage. Make it a puzzlebox with keys hidden throughout the dungeon, or levers which run underground such that no indication of what each do can be determined by just rolling and hoping. If you want to just do damage to the players at random, then do it, you are the dm, you are empowered to do so, but don't pretend like you are taking the high moral ground.
As the DM your job isn't to defeat the players. That is fairly easy to do. "Oh an Ancient Red Dragon appears." "Oh just as you defeat it, it rises as a Dracolich!" It is to create part of the interactive narrative to entertain the players and yourself. If the player is picking something like Observant, he probably doesn't care to die to random poison arrow traps, find a more interesting encounter.