DND_Reborn
The High Aldwin
There's no restriction to speaking only during your turn. So after trying to bash the door (and failing) your PC could have yelled at him to back up. If he still had movement he could. Only if he had used all his movement would he be stuck at the door.
True, but as I understand it without a house-rule or some special feature, his turn was over and he couldn't move any more even if had movement remaining.
Now, as @MarkB pointed out, if the player had realized "Hey, I failed to bust down the door, and I still have some movement left. Hey, DM, I move my last 10 fee away from the door so someone else can try. My turn is over." That would have worked and avoided the issue.
-------------------------------------
Another way would be (if I couldn't cast the spell in such a way as to avoid the other PC) to say my character can't cast Shatter without hitting the other PC so I would tell him on my turn of round 1, "Move back you idiot, I'll blow it open!". Then on round 2, he could have moved back and on my turn I would cast Shatter.
That is more how I see the current rules handling it. But, I like the house-rule that if you have movement left over you can use your reaction to move on another creature's turn. It makes more sense in the story. Imagine your character is fighting an orc and you damage him, but he is still alive at the end of your turn (you never moved at all). Another PC casts Magic Missile, killing your damaged orc. You could use your reaction to then move to engage another creature. I mean, you aren't just standing there for the rest of the 6-second turn, are you? That seems silly to me...
Instead, you glance around to see the Cleric in need of help because he has two orcs on him, but he is 45 feet away. You use your reaction after your orc is killed by the MM spell to move 30 feet towards the Cleric. Next round, you move the remaining 15 feet and can attack the orc. In the RAW, you would not have been able to move after your orc died (wasting your move), and have to move and dash the next round, not allowing you to attack.