This is an interesting topic which I feel have changed during the various D&D editions. From a game design point of view, I think D&D in 4th and 5th tried to create more balance and "control" for the DMs when planning/designing the adventures. 1st to 3rd Edition was shiftier by design, and not built for 6-8 encounters per day - more like 1-3 or 1-4 from my own experience (but you did not have the same DM support to plan your encounter design either.)
The main issue I have with both 4th and 5th Edition is that is built for 5 / 6-8 encounters per adventuring day. Which means it is very hard to challenge a party with 1 encounter per day. If a character has 40 hp and he is going to face 8 encounters, I can only deal (40/8=5) 5-8 hp/encounter in damage (if I assume they have some magical healing as well).
However, if the party are out travelling, or in the city and I want to throw a challenging random encounter at them, I suddenly have to design something that does 40 hp/encounter in damage, otherwise it will be a walk in the park for them. This is an issue that neither 4th or 5th edition handles well.
What could we do to change this? Well many things of course, but to take something from on top of my mind, I would divide HP into two parts. Stamina HP, which would be more like temporary hp in nature. You have this Stamina HP which could be maybe 10 hp ( in order to reflect the 40 hp example from above). Then we have Wound HP, which is actual physical damage. As soon as you take any wound HP you are bloodied and start to take some penalties. Maybe the Wound HP should be 5 hp in the above example. Our character now have 15 hp in total, which would allow me to throw a 15 hp/enc damage at the party. 3 times as high as the 5 hp/enc damage calculated above. And the party would be seriously challenged.
To make things interesting we can now have different healing rules.
Catch your Breath: Takes 5 minutes and allow the PC to regain up to 5 hp in Stamina HP after the encounter, if he or she took that amount of damage during the fight.
Short Rest: Takes 1 hour and allow the PC to recover all his Stamina HP, but this can only be used 2 times per day.
Extended/Long Rest: The PC recovers 1 Wound HP and all Stamina HP.
Magical Healing: The normal Cure spells, and Healing Word, etc. only recovers Stamina HP. Higher level spells as Heal is required to heal Wound HP.
This design would allow you as a DM to challenge the party with a single tough encounter, but it will not require extremely high challenge monsters to do it. It will at the same time create a possibility to throw a multiple of lesser encounters at the party and challenge them depending on the availability of time (for catching one’s breath or taking a short rest).
The exact level of the Stamina HP and Wound HP would have to be calculated based on the game effect you would like to have, as well as how to make it easy to runt with existing monsters.
/Myrhdraak