Here are very basic rules-of-thumb I use:
1) DO roll initiative when any character (or hazard, trap, or anything that uses initiative) is trying to take an action that is intended to disrupt another character's ability to take actions (for instance, by trying to kill them).
2) DON'T roll initiative simply because characters are acting in rounds. If a bunch of characters are spending move actions to move across a room and minor actions to make Perception rolls, they have no need of a random initiative order if they're all acting in sync. Initiative is, after all, a race to go first. Now, if a trap is triggered while they're doing that, go back to #1.
3) DON'T make too much out of the "he was ready long before you knew he was there" rationale. This is something I've seen time and time again. A DM gets it into his head that it's possible to be so super-duper-prepared that a character should receive some extra special benefit beyond just a chance at a surprise round. He should get an extra-long surprise round, or the enemies should have no chance of spotting the ambush with a Perception check. Many heroes exhibit an uncanny ability to react without notice against a ready foe.
As to door-clumping, I see that happen a lot too. Generally speaking, the melee-heavy parties hate it, while a mixed melee/ranged party will adore being able to keep the blasters safe and sound down the hallway. There are a lot of solutions like forced movement and lurkers popping up from the rear, but the most basic is to position enemies where the corner clips line of sight from down the hallway. You can put the melee monsters on the corners adjacent to the door rather than right in front of it. This may seem unintuitive, because they won't block entrance into the room, but I've oftened why they want to do that anyway. Instead of creating bottlenecks to hold the heroes at bay, they should be drawing them into their lair and enveloping them. Also, players love opportunities to gang up on one bad guy until he's dead; standing directly in front of the door facilitates that in spades.
