It's a smart idea. I've noticed before that there are actually relatively few encounters between levels in the great majority of games I've played. We typically only go about 3-4 sessions between levels, so that's maybe between 10-15 hours of gameplay time. Even doing something encounter intensive like a dungeon crawl, you're probably only going to see about 8-10 encounters in that time frame.
I also definitely feel like modern play skews towards infrequent but more potent and climatic encounters, which means for a lot of games you might only see 4-5 encounters before a level.
It does have knock-off effects on world-building, of course, especially for our contingent of players that use PC type builds for NPCs. I would probably state that the majority of spell-wielders in the setting are short-rest casters, probably using warlock-type mechanics. A few spells a day, bolstered by at-will invocations and lots of rituals. I would probably also have item-crafting be much more a focus; would-be wizards aren't waiting for their spells to come back, they spend their making potions, scrolls, and wands to bolster their spellcasting. Having abilities like Arcane Recovery be strictly short-rest abilities (no wait for a long rest to gain it back) could also be on the table.
I would also probably have some in-game items capable of recharging long-rest abilities at a cost, some kind of mana potion or something along those lines. I might do some house-ruling around individual long-rest abilities if I felt that some kind of regular recharge fit their fiction better. Barbarian rage is an ability I might change to maybe give a level of exhaustion, for example, the "rages per long rest" can be used to ignore exhaustion a few times before taking the long rest.