Well I'm not saying that it would be balanced with only level 1 spells or cantrips. I'm saying that as long as casters have more utility options, it won't matter how trivial they are.
Let's assume that spellcasters only get these 1st-level spells. They get undercut in combat, but they still have unique tools that can instantly solve some encounters, especially encounters outside of combat. So a giant ravine separates the party and its goal, while the wizard can no longer fly, they can send a familiar with a rope on the end. And even if you wouldn't allow that in your games, as long as one DM does and a player goes "that ruined the encounter!" Then you'll see the complaints.
Wouldn't it be better if, rather than nerfing or buffing spellcasters/martials, we gave them a different design that prevents this altogether? You could make spells more powerful and world-changing, but even combat spells might take several actions and spells with outside utility could take hours or days to cast. Like Find Familiar took 8 hours rather than 1, but they are also under the effects of whatever spell is affecting the wizard.
Or, rather than forcecage taking 1 turn to cast, it took the spell's level in rounds to cast. But once it's activated, there is no means of escape possible for its duration and it might even be a bigger cage.