"Visual [sic] evidence" isn't the only kind of perceptible. An Arcane Trickster using their invisible Mage Hand to, say, tap someone on the shoulder to give themself advantage on an attack, is generating a perceptible effect in the absence of visible evidence.
Psychic damage is damage--it says so, right on the tin--and damage is perceptible.
They don't know where you are--I don't think there's been any dispute about that--but they know something is hurting them, and they absolutely get to act, and rolling Initiative determines the order in which their actions and yours happen.