Hold it! I call shenanigans! D&D Hit points have never really measured the human body's ability to take or avoid damage well, as evidenced by being at full fighting trim until the last hp is gone. Hit Points, whatever verbiage they're dressed up in, have always been a form of ablative script immunity. They may be a little better at modelling something like endurance, but even that is a bit iffy.
Well, that's demonstrably wrong. In Chainmail, one hit was death. In OD&D, you got 1d6 hit points per level and damage was 1d6 for each weapon, the inference being each hit was actually physical damage. AD&D broke with this and inflated HP progression and then came up with the idea that HP were both, but BECMI indicated that HP were damage. 2E specifically calls it as damage only, and 3E shifted back into the nebulous area that AD&D occupied, and 4E sort of did the same, but tried to take down the physical side of HP at bloodied/half max HP. So, you're actually incorrect in the "never" point of that statement.
Furthermore, if having fewer Healing Surges in 4E but full HP (also doesn't prevent you from fighting just as hard) can be rationalized as fighting hurt, then so can being down Hit Points from your max. You simply aren't fighting at full strength after you've taken a hit. Your ability to take hits is actually less. You may swing your sword just as hard, but you are hurt as indicated by the fact that further hits may knock you out or kill you.
Presumably, there is some expenditure physical/mental/spiritual energy or endurance to either prepare or cast a spell as well.
Sure, but that's not physical damage that needs to heal overnight. If you've taken a test today on math, you can take a test on math again tomorrow with some more studying and some rest. If you take a dagger slash across your forearm, that cut is still going to be there tomorrow.
Painful wounds effect my mental concentration, to a greater or lesser degree, but maybe I'm just not tough enough.
Right, so you argument here is? If you were trying to say that when a wizard is hurt, he gains spells back slower, that might be relevant. But you're not.
I think it a valid interpretation that a purely fictional activity such as preparing and casting a spell is a little less passive than you imply here.
Except that it's not, in the rules. It takes a rest, and then some preparation (in 3E/PF it was an hour). There are no checks or anything like that, hence it is definitely passive.
I get that having a 'resource management' mini-game that expands recovery of hit points beyond a 24-hour period is a desired and fun play style for many (Personally, I think rolling your level in Hit Dice for recovered hit points after an extended rest is a good starting point), but using the word 'realism' as an argument doesn't work for me.
And I didn't use the word "realism" now did, I? And I agree with your parenthetical.
If we track spells in exactly the same fashion as we track HP, then I might be able to buy the idea that all resource management should refresh on the same time sequence. But we don't. The resources are different and come from different pools, and do different things, so why should they all be tracked the same?
There are systems that track everything the same. Systems like Mutants and Masterminds and Champions, where everything is balanced exactly on a base number of build points.
Maybe if D&D shifted completely into a spell points system we could talk about the refresh rates being synchronized, but then I'd want to see the wizards having spell points at exactly the same rate as fighters have HP. If a fighter has 40 HP, then a wizard should have 40 SP. I mean, if we're shooting for completely metagame mechanical equivalency, then let's do that. If we're going to have different systems, then I've yet to be convinced that they all need to refresh at the same rate.