How do you represent hit points as only physical harm? I'm not saying it is wrong but hit points as only physical damage is not the definition.
Well, since you asked, I'll try to explain. Bear in mind, though, that this is what I do in my campaign...it isn't going to work for everyone. But it makes sense to me.
Hit points represent the amount of physical damage your character can sustain. Burns, cuts, broken bones, that sort of thing. Without magic, it can take days or weeks to recover fully.
Non-physical types of damage (including but not limited to exhaustion, poisons, mental distress, hunger, disease, drowning, and thirst) are represented as ability damage. Depending on the circumstance, you can recover by taking a breather, applying first aid, or being bedridden for months.
So, if your character is "damaged" by poison, that is represented by a temporary penalty to its related ability score (Constitution for lethal toxins, Dexterity for paralysis, Strength for sleep, etc.) The body itself was not harmed beyond a tiny pinprick, but the potentially lethal poison inhibits the character's ability to perform the way it is supposed to.
Similarly:
Exhaustion --> Str penalty, 0 Str = collapse
Mental distress --> Wis penalty, 0 Wis = unconscious
Suffocation --> Con penalty, 0 Con = dead
Hunger --> Con penalty
Thirst --> Con penalty
and so on.
In short: if something damages the physical body, it affects the hit points. If it just affects that body's ability to work effectively, then it is ability damage.
Again, I realize that this is not for everyone. But it is how my brain works when it comes to assessing damage. After all, you don't get exhausted or bummed out by dragon fire...you get burned.