I don't think fantasy physics should be based on the speed of story. Recovery of spell slots is magical physics. The DM should rule what the rest periods are for their game not only because every sentient entity relies on understanding their world, including recharging magics, but from a game perspective, players can't plan properly if the DM keeps changing the rules at their whim.
While crossing a desert over a week, you will need your spell slots for food and water every day, deal with exhaustion every day, all that. recharging your spells and hit points once a week doesn't work.
The DM should not change the rest period on a "whim". It should be clearly communicated ahead of each adventure.
Instead, everybody should realize the mechanics need to serve the story.
It is sheer idiocy if the game insists that the character must always be refreshed and ready with all their abilities just because 24 hours has passed. (Just like how the "15 minute adventuring day" is completely needless, just because the game absolutely insists a long rest should always be once a day)
You appear to be entrenched in the false belief that the game stops working unless you can recharge every 24 hours. Nothing could be further from the truth.
You could be allowed to rechange fully (a long rest) each and every other, and the game works fine if only there is a near-continuous stream of critters thrown against you. Same with any other time duration.
All that matters is the amount of combat encounters per long rest.
If you have seven combats over 45 minutes, and then long rest for an hour...
...or you have seven combats over 10 days, and then long rest for a week...
...is functionally identical. All that matters is the number of combats per long rest.
All "it must be exactly what I'm already used to" accomplishes, is the invalidation of stories and adventures with another cadence, another frequency of combats.
The point isn't to say that one week is better or worse than 5 minutes.
The point is that the duration should vary with the story, so the same adventurer can experience many and varied adventures over her career.
Or rather, the point is that y'all can argue until you get blue in the face whether 5 minute short rests are better or worse than 60 minute short rests - you're all missing the real solution:
you are all in the right.