It does to me: how else does the poison get into me?
It doesnt neccesarily, any more than a dagger sticks in you every time a melee weapon attack scores a 'hit'.
Remember -
RAW Hit Points represent 'luck, resolve and the will to live' (in addition to health). They also represent fighting skill and experience (which is why martials get more, and you get more as you advance in experience level).
Among other things Hit Points are also 'Luck points' and 'Fighting skill and experience at avoiding blows points'.
An attack roll that scores a 'hit' against your AC might not actually strike you at all, instead being parried or dodged (using your experience and fighting skill) at the last second, or luckily glancing off armor or striking a nearby wall or door frame, or be resolutely stared down instead of striking you.
Its a question of narration.
Under "health" comes physical harm...
It does indeed, and if that's all hit points represent, you could argue this point further. They dont only represent health (that's only a fraction of what they represent, and they also represent things like luck, parrying and dodging ability, fighting skill, experience, resolve, the will to live etc) so you cant make this point.
When you read a DnD novel and read Drizzt whirling around parrying blows and dodging attacks, and blows glancing off his armor, he's losing hit points. Thats just how the 'hits' are narrated.
He has a lot of Hit Points, so he dodges and parries blows and gets lucky with attacks glancing off armor, and demonstrates resolve to keep fighting for a long time and the like constantly.
Erm...so...if the arrow never struck you at all, how did the poison on it get to you in order to force that save, particularly if you failed said save?
Maybe in some cases the arrow did glance your skin, causing a slight nick and poisoning you (HP loss represents health loss). Maybe in some cases you luckily dodged at the last moment, and the HP loss represents you tiring and your luck running out.
Its a question of narration. You choose to narrate HP loss as 'meat loss'. Thats not a fault of the system, its a fault with your imagination and interpretation of an intentionally abstract system.