So how do other people handle these situations? Do you always go with what makes things easier for the PCs? Is there a more graceful way of handling situation #2? Is there any way of having NPCs take PCs by surprise that won't leave the players shouting "That was cheap"?
In rare situations I will ask players to roll initiatives even though there is no combat happening, and I do that just to be able to deal with their actions one at a time in some kind of order. However, as a rule, if I call for initiative it is for one reason - COMBAT has commenced for one reason or another.
The FIRST individual to state they are taking a combat action is at the top of the order. If anyone else was anticipating the possibility that the individual MIGHT take an offensive action they darn well should have SAID SO. That would be the Ready action - specifically preparing a response that COULD interrupt someone else's intended action. That is, the readying character gets to go first and gets to do their own clearly stated and planned thing, and then we see if that prevents the first individual from completing their instigating action. So people who simply STATE they will do X, and people who STATE they are PLANNING to do Y if person Z does X, are the ones who go first. Everyone else rolls initiative.
Complaints? Take a number. Your initiative result determines that number. Want to be sure your character gets to do something first? Be the first to say you're doing something. Waiting to see what happens without making clear and specific plans? Luck of the dice determines your readiness to act/react - but only AFTER people who have made positive statements of action. Should it become necessary to prioritize further, then the order in which people make their statements determines the order of the priority of their actions: first-stated, first-resolved.
Surprise is a mechanic used to begin combat. If I, as DM, decide that one SIDE is surprised (the two sides always being PCs on one side, and their opponents on the other) then they are. The non-surprised side then get one free round to act while the other side gets up to speed with what's actually happening. If I, as DM, decide that neither side is really going to be caught unwares, then neither side IS caught unawares and there is no surprise resolution, and initiative alone determines order of actions. If I, as DM, decide that either side COULD be surprised, but don't care to make that determination myself (and there can be any number of reasons for that), then surprise rolls are made that will determine it instead.
Fair means of having NPC's take PC's by surprise? Same means that PC's have of taking NPC's by surprise, however, I use those means with FAR greater restraint than I ever expect PC's to do. If the DM wants to win - he wins. If the DM wants PC's to die - they die. WINNING, for a DM, is as easy as breathing. But that's not exactly a recipe for a fun, satisfying game for players to play. Can PC's be ambushed?
Of course. And if I ambush them I will have provided clues, hints or even openly stated warnings - and if players then fail to take precautions that's on them.
My aim is to never hear players shout, "That was cheap!" after being ambushed. My aim is for players to have their characters ambushed and after the battle is over shout, "That was EXCITING!"