No, no, no, no, no! There is (or should be) no such thing as a readied action outside of the initiative order. That way lies madness.
While I agree with Veg on much of what he wrote, here I have to disagree. Or rather I sort of have to disagree.
Whether we realize it or not, DMs allow actions outside of initiative all the time, so why not a readied action? I treat all situations as effectively combat rounds. As such any creature can ready an action at any time. As you indicated, the idea of aiming an arrow at a door is a readied action outside of initiative. One way to think of it is "on your initiative, you ready an arrow to fire at anyone coming through the door." Now the guy opening the door and walking through it is using his initiative to take the move actions. Bam, you fire as your ready action.
Consider spot/listen checks to begin encounters. New DMs often make the mistake of giving checks at the earliest opportunity and go from there. Invariably the players, even if they fail the checks, are alerted to something being up and begin taking metagame actions like drawing weapons. In reality, what one should be doing is adjusting the encounter set up based on the rolls. PCs roll badly? Then they're ambushed. PCs roll well? They see the ambush ahead of time.
So take the case of PCs rolling badly. The orc bowman were well aware of them for several rounds while the PCs walked into the ambush. What were the orcs doing all that time? Actions. They were using their surprise rounds to do things like draw their bows, aim, and ready.
Now reverse it. The PCs are well aware of the unaware orcs. What would you do? Well, you might go ahead and roll initiative for the players. Then according to order the players might do things like ready spells or weapons. Then every surprise round (since the orcs are unaware), the players might continue to ready or delay until someone initiates combat or the orcs become aware.
AFAIK, there's nothing in the rules that says you get one and only one surprise round. Quite frankly, if there is, I don't care, this is more reasonable and I've been doing it this way for a long time.
As veg and others pointed out, it's "how aware" is what's important.
So, let's look at the Greedo/Solo encounter from the perspective of the PC Bounty Hunter Greedo. He walks into a bar and spots Han (Han fails his spot check). Greedo, on his surprise round, decides to slip into a shadowy corner and draw a weapon. Seeing he has not been detected and not wanting to face both Han and Chewie, he takes a ready action to plug Han if he's detected (still in surprise rounds). Chewie leaves and Han still seems unaware. Knowing Jabba will pay more for Han alive, Greedo changes his readied action to intercept Han when he leaves. He intercepts Han and has the drop on him with a readied weapon. Now, at this point, both are aware and you can roll initiative. Even if Han wins, Greedo has a readied action and will automatically go first. Instead, each delays in order to talk things out (though Greedo's delay is actually a continuous ready action). However, on one of his initiative's Han decides to do a Sleight of Hand (or Bluff since he intentionally picks at the wall to distract Greedo). Greedo fails his Sense Motive and though he is 'aware' of Han, he is unaware that Han has drawn a weapon. So Han gets a surprise round, negating Greedo's readied action, and fires.
Like others, I would give some chance for the NPC to realize what was about to happen, such as a Sense Motive. Then initiative would be rolled.PCs are talking to an NPC. Both sides are obviously aware of each other. The conversation gets heated and a PC thinks that the NPC is probably going to cast a spell at them once he is done with his monologue. So not to take any chances, the PC casts a silence spell at the NPC in mid sentence.
BTW, this is why when I converse/parlay I'm always careful to draw weapons and point them at people as a readied action. (The way I do things you get a circumstance bonus on initiative.)
Now here you've actually got two different situations going on.Last session the PCs were talking to a very powerful NPC. During the talk, the PCs were being a bit obnoxious and this serious evil NPC was getting tired of their antics. So I wanted to have him cast Hypnotic Pattern just to shut them up so they know he means business. But I did not want them to think that he was casting a spell to start combat. But the 3e rules makes it hard to do that if everyone is aware, rolls initiative, and a PC beats the NPCs initiative and attacks him before he gets his spell off (thinking the NPC might be trying to harm them).
1) Normally, I give the PCs a chance to react. While I might just give them a Sense Motive or the like, usually I would do something like say "As he is speaking with you, he says 'let me show you something' and begins to reach in a bag." If no-one reacts, I give him surprise. My players learn quickly to question things: "Um, Sense Motive!" or "I draw my sword and tell him to be very careful!"
2) The players are being jerks so The DM Is Annoyed and Can Do Whatever He Wants Rule applies.
"Wait, shouldn't I get a Sense Motive or something?"
"Yep, you failed."
"But I didn't roll!"
"I rolled for you. You rolled low; he rolled high. Y'know, you should really pay attention more next time."
Really, though, most DMs just let the seriously bad guy do whatever they want as long as it doesn't really screw the PCs over. For example, in a game I've been in, we had a really big bad dude who literally was conjuring things out of nowhere, teleporting us around, and all kinds of stuff that isn't technically possible. Sure we asked for Will saves and Sense Motive checks, but we knew it didn't really matter. If it did, the DM would've given us a chance. If he didn't, then he could forget DMing us ever again.