Based on a bunch of ghost stories and examples too numerous to list, common sense would tell me this rule is wrong; and that the spectre (or whatever) can move through other creatures and objects as if those obstacles were not present.
That they take damage if they remain inside an object seems to indicate considerably more interaction with the "real" game world than I'd have expected. That said, there's design space for both: some incorporeal creatures might move freely through obstacles as a creature ability while others have to treat them as difficult terrain etc.
Well, the idea is that while the ghost is actually incorporeal--as in, it is truly manifested on the Material Plane, with its ghostly body--then it really
is interacting with the, as you put it, "'real' game world" to a greater degree. If it instead used its Etherealness ability to shift to the Border Ethereal, and then employed its Ethereal Sight to look up to 60' around itself into the Material Plane, then it would not
actually be passing through those objects--it would simply be located at the space in the Ethereal where that object happens to be in the Material.
It's similar to, but of course not identical to, the difference between using
plane shift to go to another plane, vs using some kind of scrying spell to merely view the place that corresponds to your current location in a parallel plane. If you
plane shift, your body is actually there and has to respect the objects around it. If you simply scry, then the only thing that matters is what objects are near you in your current plane, not the ones in the other plane.
This reasoning makes sense to me based solely on what's written. I suspect, thinking quickly about it, the rules are written as they are mostly to prevent PC-side abuse of effects that make them ethereal.
Almost certainly.
The vast majority of rules are written in a player-facing way, because the player is the one actually using most of those rules. Anything not actually facing the players, so long as it is self-consistent and reasonable, need not bear any similarity at all to player-facing rules. This is not merely a game design convenience, it is the recognition that rules designed for player use usually need to be tighter, cleaner, and more specific, because misuse or abuse is a bigger long-term concern. A monster lasts, what, five rounds? If that, in most cases! A player-facing rule about tremorsense lasts for
the entire game's existence. There's less pressure (not no pressure! Just
less pressure) to do things Just Right with the monster's rules than there is with the rules actually visible to the players.