Sandbox games in 3e and 4e have one issue in common that isn't necessarily present in sandbox games using older editions of D&D - looking at a foe doesn't give the players a clue whether the encounter is easy or hard.
In AD&D, if you saw an encampment of 300 orcs or a huge dragon, a big red flag was waving in your face to run. Likewise if you saw one orc, you didn't presume that he might be 20th level with 200 hp. If 5 goblins saw a fighter in platemail, they would suspect he could plow through them in one round. In all these cases, the likelihood of parley increased and the likelihood of combat decreased.
In later editions, the appearance of a creature can be very deceiving - it can be very easy or very hard. Character levels, HD advancement, reskinning, etc. gives the DM too many tools to make the encounter unpredictable. In a sandbox game, unpredictability is fine when the encounter is level-appropriate. Not so much when the DM wants to simulate a world where encounters can either be too low or too high relative to the level of the PC's. If players have no idea how hard or easy an encounter might be, their first instinct is to test it through combat.
IMO, the key to these sandbox games is bring back the red flag, both for the PC's and the NPC's. If they know in round one that there are huge negative consequences to combat, the roleplaying begins and the flavor of the sandbox is preserved without wasting time with unfun activities. It won't work in every case - the PC's still have the right to attack and mindless NPC's won't know any better.
How to bring back the red flag? House rule Sense Motive/Insight checks that give an indication of the combat power of an opponent. The larger the discrepancy in CR/levels, the easier the check becomes. Liberal use of this check by NPC's will tell them when they need to run or not start a fight in the first place. Encouraging PC's to use it should achieve the same results.
If players know that a 'dire threat' will result in either TPK or boring combat, they will be thankful to have the opportunity to do something else more creative. It might be a little metagamey or too close to a MMORPG (I know Everquest has this mechanic, I have never played WoW), but it's the best approach I can think of to solve the problem.
Note I personally have not playtested any specific houserule that incorporates this idea, but I'm pretty sure others here have experimented with this and can comment on it. I do plan to use it in my next campaign, which will be a 3.75 sandbox.
In AD&D, if you saw an encampment of 300 orcs or a huge dragon, a big red flag was waving in your face to run. Likewise if you saw one orc, you didn't presume that he might be 20th level with 200 hp. If 5 goblins saw a fighter in platemail, they would suspect he could plow through them in one round. In all these cases, the likelihood of parley increased and the likelihood of combat decreased.
In later editions, the appearance of a creature can be very deceiving - it can be very easy or very hard. Character levels, HD advancement, reskinning, etc. gives the DM too many tools to make the encounter unpredictable. In a sandbox game, unpredictability is fine when the encounter is level-appropriate. Not so much when the DM wants to simulate a world where encounters can either be too low or too high relative to the level of the PC's. If players have no idea how hard or easy an encounter might be, their first instinct is to test it through combat.
IMO, the key to these sandbox games is bring back the red flag, both for the PC's and the NPC's. If they know in round one that there are huge negative consequences to combat, the roleplaying begins and the flavor of the sandbox is preserved without wasting time with unfun activities. It won't work in every case - the PC's still have the right to attack and mindless NPC's won't know any better.
How to bring back the red flag? House rule Sense Motive/Insight checks that give an indication of the combat power of an opponent. The larger the discrepancy in CR/levels, the easier the check becomes. Liberal use of this check by NPC's will tell them when they need to run or not start a fight in the first place. Encouraging PC's to use it should achieve the same results.
If players know that a 'dire threat' will result in either TPK or boring combat, they will be thankful to have the opportunity to do something else more creative. It might be a little metagamey or too close to a MMORPG (I know Everquest has this mechanic, I have never played WoW), but it's the best approach I can think of to solve the problem.
Note I personally have not playtested any specific houserule that incorporates this idea, but I'm pretty sure others here have experimented with this and can comment on it. I do plan to use it in my next campaign, which will be a 3.75 sandbox.