There are times when it won't work for stealth, too. Like, if the DM decides that a guard who spots a hiding character is going to pretend like he doesn't, that stealth check in the moment will give away more information than you want to.
So those are the times when I'd go to the guard rolling...