Hriston
Dungeon Master of Middle-earth (He/him)
Let's say you think there's a Shadow Monk hiding behind a rock, getting ready to kill you. There isn't, but there's an assassin aiming a bow at you from the second-floor window. In this scene, you cannot be surprised even though the threat you thought you imagined was illusory. In fact, you may never even know that the assassin who fired the bow at you wasn't the same person as the Shadow Monk you had deduced to be present earlier; as far as you're concerned, he probably just teleported up to the window right before firing at you. The only thing that matters as far as surprise is concerned is whether you "notice a threat," or as I've called it, are "in kill mode."
I'm not sure about all the non-bolded stuff you have. It sounds like you're saying you think the PC would be surprised in this scenario because the threat wasn't where he expected it to be, but that's not how I would rule it. At best that's just an attack from an unseen attacker, which grants advantage to the attacker.
Imagining a threat is not the same as noticing a threat. You actually can't notice something that isn't really there because noticing requires verifying the presence of something with your senses. You can't sense something that isn't there. Even if you think you have, you haven't, not really. The surprise rule doesn't say that someone who doesn't imagine a threat, or doesn't think they've noticed a threat is surprised. No, it actually says that someone who doesn't notice a real threat is surprised. I think your interpretation that surprise is caused by the lack of a certain mental state is leading you to disregard the rule in cases in which it would clearly apply. Of course, you are free to do this, but all of this has given me an idea:
A character flaw that gives a character immunity to surprise because the character has delusions of constantly being hunted by an imaginary pursuer, that he is convinced is around every corner. Regularly incurring some level of exhaustion seems like a reasonable trade-off.
Also, what of mixed-alignment parties in which all members are not to be entirely trusted? Such characters could very reasonably be construed as a potential threat, which would require other party members to watch their backs. Would having such members on board keep the party from being surprised?