I think that is a perfectly legitmate way of viewing hit points. Even if your not hit, muscles get pulled, bruises, scrapes, minor lacerations, etc. start adding up and slowing you down. This stuff represents "injury" maybe not deep wounds but "damage" that needs time to heal none the less...