When the hidden Rogue says 'I shoot the guard with my crossbow' he's declaring an action then and there. As soon as the dice get thrown down to resolve that action, and initiative determined, he cant 'take it back' now that he rolled poorly on that Dexterity ability check to sequence that action.
He's committed now.
That's a generally reasonable house rule, but very much a house rule. Per RAW, you declare your action on your turn, not before it. And until initiative is rolled, it can't be your turn because there are no turns.
This is one of the many hiccups that result from having "roll initiative" a) be a discrete event with mechanical consequences, and b) take place
before the action that triggers an initiative roll. My group has stumbled over this little paradox more than once. I hope the next edition, whenever it comes, does away with a). Rolling initiative should be viewed simply as determining a pre-existing state of affairs: You are
always in initiative order, but normally you don't care what that order is. As soon as it becomes important to know, you roll initiative.
In this approach, surprise would not depend on "Initiative was just rolled"--it would be the result of "On my last turn, I was not aware that combat was imminent, and since then combat has begun." If the action that starts combat is an assassin attacking an unaware target, the assassin gets the benefit of surprise, end of story. If the assassin sees the result of the initiative roll and for some reason decides not to attack, okay. You don't attack. Nothing happens. If the initiative roll doesn't change anything, it doesn't matter if it's followed by combat or not.