I wasn't sure how this could happen, or why a DM would do this. It sounds like you're talking about a situation where both parties are surprised. I haven't said how I'd handle this situation if it ever comes up, although with surprise being as difficult to get as it is in 5E, I don't think it would be very often. At best there's usually only a few people on each side that are surprised in an encounter. Then I looked at the surprise rules in the SRD. It has an odd way of dealing with both sides being unaware. It says you skip the surprise round and move on to normal initiative. If both sides are unaware of each other, I'm not sure what's supposed to happen after that. It seems the designers were determined for there to be an encounter anyway.
This is because initiative isn't rolled until someone declares a hostile action. You aren't rolling for initiative until there IS a combat because one side or the other IS attacking. Technically, if you open a door, "surprise" the creatures behind it and they are also surprised because they didn't know you were coming in...you still don't roll initiative if neither side has any intention of attacking each other. You just run it as a role playing encounter.
Which is precisely why I get annoyed when some DMs decide to run a surprise round where nobody attacks. Or to skip the surprise round when there should obviously be one and put hidden or invisible enemies into the initiative without taking any attacks.
It leads to situations where the DM has everyone roll for initiative and then has the "hidden" enemies take non-violent actions that allow them to remain hidden while the PCs are confused as to why they even rolled initiative in the first place. It is frustrating to be told "Alright, it's your turn. What do you do?" when there is literally ZERO reason to take an action that isn't "Well, I was walking down the corridor before, so I keep walking down the corridor. Nothing's changed, right?"
The proper way to do this is that the enemies continue to follow the PCs or get into the proper positions to attack while completely out of combat. Then, when the first one makes an attack of some sort, you roll for initiative and give the enemies a free round of attacks(the surprise round) before asking the PCs what they do.
This is very different from the situation in 5E where you can begin an encounter from hiding. If your opponents are completely unaware of you, you may have the opportunity to avoid a fight altogether while your opponents are surprised. If both sides are surprised, they may not, in fact, ever notice each other at all, and strict adherence to the rules would have you running a first round in which no one can do much of anything. I agree that rolling initiative and moving on to round one makes little sense in this scenario. I would most likely just tell the players they don't notice anyone, but then there's the issue of what happens if they stop hiding and give their location away to the other side. I'm beginning to think that the best way to handle it would be to ignore the encounter entirely. The two parties simply sneak past each other without alerting one another to their presence. I seem to recall that the playtest had some language to that effect.
An encounter doesn't happen unless one side actually ENCOUNTERS the other side. At least one side has to be aware of the other side for an encounter to start at all. And initiative isn't rolled until the moment BOTH sides become aware of each other(or, more accurately, milliseconds BEFORE both sides become aware of each other).
What if a hostile action is what makes one party aware of the other? Shouldn't you roll initiative before that action is declared, or just before it is resolved?
Yes. As I said, you roll initiative when that action is declared(but before it actually takes place). Because now you know with 100% certainty that a battle is going to take place because an attack has been declared.
Then you determine surprise(the other side doesn't know the attack is going to take place, since they don't see the enemy, so there is surprise). Then you roll for initiative and everyone on one side of the combat goes in initiative order taking actions...at least one of which was the formerly declared attack. Then it goes to the first round of combat and both sides act in initiative order.
Shouldn't initiative be rolled before this to adjudicate things like when a creature is no longer surprised and can take reactions?
Yes, that's what happens. One side is unseen so they have surprise. You roll for initiative and they get to act during the surprise round. So you go down from the highest initiative down to the lowest initiative of all the enemies and they each get a round of actions while the other side(the people who are surprised) don't get to take actions and can't make reactions. Until you get to the lowest initiative in the surprise round and then start back at the highest initiative during round "1". Then from that point onward no one is "surprised" anymore.
I absolutely agree with all of this, except the last part. I only award surprise when it's the result of an attempt to be stealthy, not for other surprising events.
I try to avoid it whenever possible since, as I said before, it's a little unfair to allow someone to attack while perfectly visible without given fast PCs(or fast monsters) the chance to react. But there are certain situations where the idea that someone would "react" to something they never saw coming is kind of silly. The book says that you need cover or concealment to successfully hide. Which makes sense, since if someone can see you, you aren't exactly hidden. But it also says that the DM can rule in certain circumstances while out of combat that a creature is looking in a different direction or distracted by something and therefore you don't need cover or concealment to hide from them. In combat, it's assumed people are always paying enough attention to avoid this.
I just extend this to its logical conclusion. The spirit of it being that if the creature in question doesn't "see it coming" you can hide from them and if you are hidden from someone when the encounter starts, you've successfully surprised the other side. It's certainly up to each DM to interpret this their own way but I've allowed people deception checks to seem completely non threatening while allowing slight of hand checks to draw small weapons without anyone seeing and then allowed someone to get a surprise round. I almost never allow any of their friends to join in on that surprise round since they were just as unaware of the attack as everyone else, though.