Your ruling at the table was fine. It's the DM's job to keep the action moving, and I tend to err on the side of caution on things like this myself.
As far as official rule, the closest I can find as far as rules is under
Casting a Spell in the PHB:
If no unblocked straight line extends from the point of origin to a location within the area of effect, that location isn't included in the spell's area.
So if the room isn't big enough, whatever the obstruction is becomes one edge of the spell. A closed door might be blocked if it opens in. A door that opens out would cause an opening when the door is open.