You can take 20 in Spot, its just impractical. It's just looking for 2 minutes straight. There's no penalty for not seeing someone, and Spot specifies that you can Try Again. So as long as they don't interrupt you (and its in their interest to do so, unless they aren't trying to ambush and are trying to evade notice, in the case of the lone rogue hiding from a band of warriors), you can keep looking.
Personally, this is why I'm in favor of the One Opposed Roll; it avoids the tedium of one guy taking 20 and five guys rolling constantly to see him, or both taking 20 and comparing skill ranks (at which point its higher score wins, and there's no random chance involved, just circumstance modifiers).
Which is also the problem of five guys trying to ambush five guys, that's 10 d20's being rolled. Someone on the ambushers is going to rathole the roll and give away their position, and someone on the Spotting team is going to roll great and see everything. You're better off rolling a d20 for each side, otherwise the ambushers (skill levels being roughly equal) will never succeed.
Edit for clarification: Taking 20 on a Spot check. Its not something you can do all the time, but if the group was anticipating ambush, that's how I'd do it. Take 20 on a Spot, move to the next covered position, rinse, repeat.