The assumption that spell caster will use their limited resources instead of their limitless resources while rationing limited resources is just that -- an assumption. If that's what happens in your experience and not what happens in my experience then I know that it's just an assumption simply for the fact I know it doesn't happen at all tables. If it doesn't happen at all tables then it's how the players are playing and how the DM is handling it, not a proven issue with the rules themselves.
That some tables don't experience an issue that others are experiencing doesn't mean it's not an issue with the rules.
Nor have you even shown it's not an issue at those other tables - all you have done is asserted that's the truth. It's always possible that the issue be present and no at a particular table cares enough to notice it. It's possible that something has been the status quo so long that you wouldn't know that your status quo is exactly the thing I'm calling an issue. - I'm not saying that's necessarily the case, just that you are also making an assumption that there's tables where this isn't a problem.
Then even assuming that your assumption is true - that some tables actually don't have this issue - it's quite possible and likely IMO that they way they are avoiding it is by some means that my table would dislike and/or wouldn't work for the specific playstyles the players at another table have.
Thus, the explanation is that it's the rules fault - because if the rules are going to allow and encourage multiple ways to play the game then all ways to play the game should avoid the big issues.
What happens is relative to the adventure and campaign.
Well, there's the example of the goblins who weren't defeated yet tracking down the party before they've managed to complete their rest, for starters.
Which only incentivizes players to rest even more often with even more resources remaining in case something like this happens again. You've arguably just made the problem worse.
If you pull something as cliché as "caravan guard", cultists attack the caravan, you blow daily resources fighting them off, and group "b" was kidnapping the caravan master's daughter (the plot hook) while you fighting the other attackers do you really thing the appropriate response to "save my daughter" is "I'll do it tomorrow".
Caravan master: "Umm, why tomorrow? Save my daughter! I hired you to protect us!"
Players: "Well, you see, Bob got excited and was trying to show off, and now he wants to relax so he can show off again tomorrow."
Caravan master: "So I hired Bob and he's totally useless after one battle? Any you others are helpless without him?"
Players: "Well, no. We never actually needed Bob and none us is helpless at all. Not even Bob."
Caravan master: "So really what's going on here is you are just leaving my daughter to whatever happens to her with the those cultists....."
Some players/characters would respond "if you want our help getting her back then you'd better let us do it our way - or we may just decide to go find something better to do"
Going with the not being out of spells the party starts the adventure, decides to quit for the day early (not necessarily right after the caravan fight but at any point), and the caravan master's daughter becomes the victim of ritual sacrifice. The goal is to save the daughter. The party chose to play in such a way that they fail the quest.
And they will never know that the ritual wasn't destined to be finished before they arrive. If you want players to play toward a timeframe you better make dang sure they know what that timeframe is. Some insinuated "get there before it's too late timeframe" rings hollow in a game where getting there in a weakened state can just as easily mean you fail to save the girl and get yourself killed in the process.
took the approach of gathering forces. If the party wasn't progressing through the dungeon there would be replacements for what they already went through and increasing numbers of opponents by their inaction.
Which is evidence of the problem - You've got to actively write the ficitional adventures in such a way that they always conform to the resting period.
24 hours is enough to complete that ancient ritual summoning ancient evil, important NPC's to become compromised or slain, villages to be razed, and more. It's plenty of time for the remaining forces to counter attack, shore of defenses, recruit additional resources, set traps, relocate important resources, and set extra patrols.
Not always. Some rituals take weeks or months. Sometimes important NPC's get interrogated for weeks before death. Getting forces to villages is typically a more than 24 hour task. 24 hours isn't enough time for reinforcements in most cases - if there even are reinrforcements to be had.
The 6-8 encounter guidelines are an estimate to when the party should need rest. The long rest is the party sleeping overnight. It doesn't makes sense outside of a gamist approach that the party goes on an adventure and does nothing but wait until nightfall so they can sleep again before continuing, and events certainly progress while the party is doing nothing but waiting so they can sleep again.
Characters have a personal survival instinct... It's all in fiction and nothing gamist about that.
The only thing a DM needs to do in that scenario is add encounters to it because the party is literally doing nothing in that time as well. That's why it's actually easier to take a short rest than it is to wait to be able to take a long rest again. The party already benefitted from a short rest well within 24 hours prior and doesn't actually start the long rest for some time.
Encounters don't make sense to add in every scenario.
By the way I think I realize why you don't have 5MWD's. It's because you don't DM toward the fiction but take the gamist approach of always making the fiction conform to the games recovery mechanisms. That is you eliminate them by removing a large amount of possible fictional outcomes - which for me is something I'm not willing to do and why your solution won't work for me.
Even the party that decides to take a long rest benefits from a short rest an hour into the wait. At that point they aren't out of resources again because they spent hit dice and most classes either didn't need to recover anything or recovered something.
That's a very debatable point. Many play that long rests and short rests are more mutually exclusive.
