DM: What do you do?
PC: I attack the nearest bugbear.
DM: Where is that?
PC: Wherever I hear it.
DM: You hear three creatures near you, fighting. You have no idea which is an ally, and which is a bugbear. They're not hidden, you just don't know which is which.
This is how illusions work as well by the way. By your logic, if something appears to be something else, it doesn't matter, you identify what it is purely by hearing. Which is...nonsense. Your logic also says darkvision is useless, and light sources are useless, most of the time. There's just nothing at the table that works about the way you guys are reading the rules. Your hearing is not a substitute for vision.