Why not have the speed of recovery related to the spell? After *spell level* hours you regain the spell. Or after those hours of rest (not always sleep).
I think you woul;d greatly increase the power of all casters. Clerics should be able to heal any wounds the party has with ease giving time. Magic would become a much bigger part of the world as well.
It might be a little bit of a pain keeping track of the time though.
"Okay I just cast my last fireball at 3:12 pm, but I also cast that haste at 12:22 so I should be getting that back in 10 minutes. When did I cast those magic missiles again?? "
Good point, I was thinking unless you play very combat orientated, you would have stopped anyway for long enough that to be irrlevant. I never thought of multiple spells of the same level, damn. Still it was only a random idea.
The longer people stop to wait for the spell casters to get back spells, the worse it can be if they are attacked. Random encounters are meant to hit the PCs when they aren't really ready.