That's because if you know where they are (and are objectively correct in that knowledge, as determined by the DM) they are not hidden from you.
They cant take the Hide action if you saw them go behind the pillar. You saw them go into hiding. The rule is not 'You cant [take the Hide action] when you are being watched;' the rule is 'you cant [attempt to hide] when someone is aware of you and is watching you'.
Youre parsing the Hide action as thing in and of itself. The RAW doesnt do this. Its just a plain english statement of common sense.
Look, come to my house and 'hide' from me in my bedroom while I watch you crawl into your hiding spot (your choice of under my bed, the closet or behind the bookcase). We can repeat this experiment as often as we want, but I assure you at no stage will you be hidden from me (despite having total cover relative to me).
After a dozen or so attempts, lets try it again, only this time, I wont be observing you (Ill close my eyes and turn up the stereo), and you can hide anywhere in the room you want.
Its not that you're doing anything different each time when you Hide, but when I watch you go into hiding it's impossible for you to do so. When I'm not observing you go into hiding, it suddenly becomes possible.
And if they did teleport away after moving behind the pillar, then I would allow the Hide action to be taken (the observer was not watching them go into hiding in this case).
Its a general rule. If you see someone going into a hiding spot, they cant hide from you there. They might have total cover relative to you, but they arent hidden (in game terms they simply cannot take the Hide action - relative to you - once there).
Mate, if you know where someone or something is, and are objectively correct in that knowledge, then (even though it may have total cover relative to you) it is not (objectively speaking) hidden from you.
Total cover, heavy obscurement and even invisibility does not = hidden.
Thats not a houserule. If a creature is hidden it can lean around the pillar and make an attack with the benefits of being hidden. After the attack is resolved (succesful or otherwise) the creature reveals itself and is no longer hidden (and generally cannot again become hidden as long as its being watched).
You walk into a room with a rogue hidden behind a pillar in the room. He pops up and shoots you gaining all the advantage of being hidden (advantage on the attack roll and sneak attack) and after this attack is resolved, he is no longer hidden.
From this point onwards the game assumes that you are now aware of him, and he cant generally attempt to hide again. Even if he ducks back behind the pillar, it doesnt matter (although he does gain total cover). You saw where he went, and (while he has total cover relative to you) barring some kind of extreme outlier (he teleports away behind the pillar behind you) he cant attempt the Hide action once behind the pillar again.
No they cant (barring having the Skulker feat or being a Wood elf). They need heavy obscurement to hide, not shadows, and in any event if I am watching them go into their hiding spot, they cant attempt the Hide action when there.