From a design standpoint, I don't really see a
need for both AC and hit points. Let's examine two possible systems, and compare how they work. Bear in mind that both of these systems are drastically different from core D&D, and if we actually used them, many standards of the game would have to be changed.
System One - Defense Only: In this system, your character gets Wound Points (WP) equal to his Constitution (or 1/2 Con for Small characters). You never get hit points, and WP heal at a rate of 1 per day of rest. When you reach 0 WP, you are disabled, and must make a Fortitude save (DC 10) every minute to keep from taking another point of damage. If you roll well enough, you can stabilize, but if you fail your Fortitude save, you'll slowly keep taking damage. At -10 WP, you die. While at negative hit points, you'll usually still be conscious and can take a partial action each round, which allows for dying speeches or daring last stands.
Your base AC is equal to 10 + Dex + Base Attack Bonus + Armor and Shield + various miscellaneous enhancements. AC represents how hard it is to get a blow through that strikes with any significant force. Beating AC might represent slipping a dagger through armor, or slamming a mace hard enough to crack bone even through plates of defense.
This system attempts to model reality. As you get more experienced, you get better at dodging blows, and you can use armor to protect yourself, but if you're hit, you go down very quickly. Spell damage would have to be drastically reduced, so a fireball might just deal 1d6 points of fire damage to everyone in the area. Similarly, things like Sneak Attack and Power Attack would have to be scaled
way down.
The drawback of this system is that you don't get to roll as many dice, and everyone's going to want AC-boosting and damage reduction magic items above pretty much anything else.
System Two - Diverse: In this system, we divide up different types of defense into different systems.
- Dodging attacks: Hit Points.
- Surviving injury: Wound Points.
- Parrying incoming attacks (Combat Expertise, Shields): Armor Class.
- Passive Defenses (Armor, Cover): Armor Class.
Dexterity no longer modifies your defenses (it still helps with Initiative, Reflex saves, and Tumble checks to avoid Attacks of Opportunity, so it's not useless). No longer will 'nimble rogues' be the masters of surviving combat. Fighters are supposed to be the ones who survive combat, so high hit points can represent a nimble fighting style for a fencer, or a very tough body for a tank. You can still have nimble rogues, good at dodging spells and traps, but an equal-level fighter will take them to pieces in close combat.
You heal at a rate of 1 HP per level per hour. WP work like detailed above.
My Personal House Rules: I don't like giving out magic armor. I mean, do all high-level bad guys have magic armor? If so, then after a few fights, the PCs would have dozens of suits of magical armor, and I don't like people having throwaway magic.
Thus, I get rid of the assumption that everyone has magic at high levels. Potions and scrolls and other limited-use things I'm fine with, but magic weapons are rare and unique, and magic armor and other defenses are practically non-existent. To keep PCs and NPCs fairly balanced, though, everyone gets a Defense bonus to AC equal to half their character level. Simple.