Well yeah, obviously, you want to avoid the 5-minute workday. But at the highest levels, it becomes more that you don't tell the players what adventures they are going on, they tell you. With the ability to teleport, plane shift, jump into an extradimensional mansion, or even pop off to one's own private demiplane where they control the flow of time, running a high level spellcaster out of resources requires pretty much ad hoc cheating via a deus ex machina.
When you can use powerful Divinations to know what your target is doing, then materialize a fully buffed part of adventurers into their bedroom, the game turns into an all out wizard war, and the DM quickly gets locked into an arms race to constantly try to challenge characters who can create armies of Shadesteel Golems, Simulacrum Artificers to craft any magic items they wish to have, and Infinite Wishes to pay for it all.