These usually boil down to judgment calls.
Others have raised the point that a lot of high-level spells don't actually do anything truly scope-changing (IIRC a conversation with FormerlyHemlock on another board where the goal was to stop a slow-moving lava-flow from wiping out a town, and the end-realization was that pretty much all of the environment-changing/battlefield-control spells that casters have are woefully insufficient to address anything other than other small groups of skirmish-level combatants). This is true of most spells -- they may have outsized impact on game-typical combat encounters (and perhaps should be adjusted for overall balance), but they don't actually let the casters do new things that everyone else can't*. The exceptions are thing like the aforementioned teleports, plane shifts, resurrections, and the one I see the most (perhaps because it is lower level) -- fly. You can make it concentration-only and limited duration and such; change the fighter to be as folklore-mythic in their climbing or jumping abilities as you want (so long as they can only leap a certain height, as opposed to 'as high as plot demands'); but if the DM puts the lever that lets the party into the next wing of the dungeon across a bottomless pit, or throws in a flying castle 5' above that max jump height, suddenly it is a problem that the caster can solve that the fighter can't**, and for historical reasons people expect it to show up when casters hit level 5.
*a significantly more common scenario is simply the magic way of accomplishing task X is automatic after spending the slot (and lots of groups having trouble making limited spell slots more than a paper-tiger of a constraint after a certain point), the magnitude of the effect outsized, or the limitations not being very limiting.
**natively. Obviously magic items change this.
This, I think, is one of the big issues. AD&D added limitations in terms of spell-fizzling (but some durability bonuses in the form of bracers or armor and some new defensive spells), but otherwise each edition has generally removed or reduced (in consequence) an inconvenience that casters previously had (be that spell fizzling, or atrocious AC/HP, or limits on weapons usable, or the damage you could do with them or the magic abilities you would get with the weapons you were allowed, or for 5e the removing of having to prepare the exact loadout of spells you think you will want to cast in a given day, or so on...). Oftentimes this was done for good reasons (balancing powerful with 'this is really inconvenient' and/or 'but you are utterly worthless when not doing this' both being a form of balance a lot of people didn't really enjoy), but it's interesting that no new limitations were added in their place, nor were the other classes really given any boosts nearly as situation-changing*.
*generally instead little boosts around the edges. No one would say that Action Surge is trivial, but id doesn't open up new avenues of playstyle previous constraints meant for fighters, or the like.