Hmmm...if a lot of people are consistently getting it wrong, might that not suggest that it is, in fact, a system problem? Or at least a communication one?
I tend to agree that perception is overly broad, and making it a skill plus tying it to wisdom makes for some weird story dynamics. Like, it always feels off to me that the bookish Cleric is the party's go-to person for sensing danger in the wilderness, while the barbarian who has spent a lifetime in the wild is terrible at hearing creatures "moving stealthily in the forest" (PHB).
And the fact that it is so widely used speaks to it being something that probably every character should just have, with perhaps a bonus coming from their background via a feat, not their ability scores.