Replace short rest with a memento system.
Your background and race gives you two mementos. These can be trinkets, memories, or markings that you cherish about your old life, whatever that may be. If you spend 1 minute cherishing your memento, you remember why you're fighting/adventuring in the first place, and gain a boost in resolve to keep going forward. This gives you the benefits of a short rest.
You can do this twice. You regain expended uses when you finish a long rest. This is because you won't get much out of a cherished thing if you spend every waking moment appreciating it, but when you're really beat down and you remember the weight of your father's toy dagger around your neck, the strength to keep on fighting returns to you.
This provides a narrative reason for short rests to be taken by the players whenever they want, and it lets you reflavor this reason in a thousand different ways, from godly boon, to inherently magic world, etc etc etc.
Then you can keep Pact Magic as is.