You're reading RAW, not RAI.
As far as I can tell, I'm doing both. You don't happen to like RAW in this case, so you are claiming that it is not RAI. That doesn't make it so.
Such an interpretation makes that situation doable, but such a situation is definitely unfun and unfair.
Possibly. What is preventing the PCs from just walking away? Must foes always be killed in order for there to be fun?
First of all, it says that creatures occupy squares, but it doesn't actually say that large objects can't occupy squares. You can't take that to mean objects don't, just as you can't assume that, for example, elves aren't smart because you know for a fact that all eladrin are smart. Just because all eladrin are doesn't mean that all elves can't be either; just because all creatures occupy squares doesn't mean that all objects can't either.
Occupied Squares is a game mechanics term and there is an entire section on it called, curiously enough, "Occupied Squares". So yes, creatures occupy squares, objects do not in the game system.
Second, if a wall takes up the entire square and has a shelf on it, technically the shelf is in the square that the creature is in when next to the shelf. Small objects like that and dropped items do not occupy the square; they are too tiny. Medium objects and larger, I would say, occupy squares. Small creatures can occupy squares since they are moving around, but Small objects (like certain types of difficult terrain) are still and so a creature can enter that space. Tiny creatures don't even really occupy spaces until there's four of them in the same space; why would Tiny objects?
Who said that a wall must be contained within an entire square in order to be blocking terrain? You are working on a faulty premise here.
Third, if a phasing creature can end its move in a space with a wall (hypothetically assuming that blocking terrain does not count as occupied squares), how does it make sense that it can't end its move in a space with a creature, which takes up less room and is probably easier to go through?
It doesn't have to make sense. It's the rule. There are literally dozens of rules that don't make sense.
I understand that you do not like the scenario as presented, but that doesn't mean that the DM did not follow RAW. As far as I can tell, he did. This just happens to be a scenario that the designers did not envision ahead of time. It happens.