Yeah, I don't think there are any rules concerning this. I can't see why not, as almost any dungeon with 5' corridors can run into the problem of fighting critters in close quarter hallways.
And what about all those times when you have a bunch of swashbucklers drop on the party in a tight spot? Or when an absurd number of kobolds is holding the party off at a corridor opening? Or when the front line fighters are buying time for the wizard to find his wand of lightning bolts? Not that it comes up all that often, but it isn't covered in the rules in any detail.
So I houseruled it.
1. Any PC weilding a two-handed weapon or Large weapons cannot use it while another PC is sharing the 5' square. If the weapon is weilded, there is a 50% chance the other occupant is hit.
2. PCs in the same 5' square take a -6 penalty to attack and -2 to AC. Feats involving coordination of attacks/defense negates the penalties. If
3. Any enemies attacking the 5' square and missing still has a 20% chance to hit the other occupant of the 5' square.
4. No casting or use of magic items (other than potions) is allowed.
5. Occupants must take a full round action to draw a weapon safely or pull something from their backpacks. An occupant may use only a move equivalent action to draw a weapon, but has a 50% chance of hitting the other occupant. Quickdraw negates all of the above penalties.
6. Entering into an opponent's 5' square provokes an AoO at +4 unless using tumble, bull rush, or another method of entering the opponent's 5' square.
7. If one occupant wishes to grapple the other, there is no AoO, and the grapple check is given a +4 to the aggressor.
8. Tripping against the other occupant provokes no AoO, and the attacker is given a +4 bonus.
9. Moving out of a shared space provokes an AoO, and only one AoO can be made against an opponent that is leaving your space and moving out of your threatened range. A 5' step out of the shared space also provokes an AoO.
10. Opponents that are occupying the same 5' square are automatically flanked if there is another opponent in any surrounding square.
11. If one or both of the creatures is small in size, the penalties are decreased by 2. If one of the creatures is large in size, the penalties are increased by 2. Any size catagory over large cannot share a 5' space.
Sure those rules don't cover every nuance, but they are a start, and I intend to keep and use them for whenever I start DMing again. I added the rules to be a potential pain for the PCs to use, as I don't really advocate the idea of everyone sharing 5' squares. I think it is necessary though, especially when you have a house, tavern or dungeon crawling with dozens of critters that are squeezing into every corner.
Now the problem is using it with larger monsters and making sense to the players about why they cannot share a single 5' square with a huge dragon.
Ok, I'm done, I'll go back to the House rules forum now.
