Changing the rules you've set up about the Astral planes seems pretty petty and abusive in of itself. The group will be very unhappy if you do this. I'd avoid it if I were you. However, since they do seem to be abusing it....
Here's the plane shift spell as writ:
You move yourself or some other creature to another plane of existence or alternate dimension. If several willing persons link hands in a circle, as many as eight can be affected by the plane shift at the same time. Precise accuracy as to a particular arrival location on the intended plane is nigh impossible. From the Material Plane, you can reach any other plane, though you appear 5 to 500 miles (5d%) from your intended destination.
Now, what I'd rule is that due to subtle shifts in the astral plane, you may or may not wind up at your intended destination. I wouldn't rule that they'd wind up between walls or something, but just off course. That's one way of dealing with this.