I totally am sick of hearing it too. Everybody gets to improvise, but the casters get these chunk of mechanic wih a CLEAR and PRECISE effect that are far more reliable than "I roll for Acrobatics".
This is the major difference.
Skills, improvisational actions and maneuvers that can be done at will are compared to what are supposed to be limited use clear precise mechanics (spells). This is a sensical approach as long as it is balanced correctly.
If casters are using spells as often as martials are improvising then, either the adventure is poorly designed or the game is poorly designed.
5E and modern D&D gives spell casters so much more opportunity to use magic. Compare a 1st level B/X magic user who gets 1 spell per day vs a 1st level 5E Wizard who can cast 3 spells per day (2+Arcane Recovery) + unlimited cantrips (which on their own are better than most non spellcaster actions).
In B/X the Magic User only has one time per game where they can automatically inject their agency by way of a rule (spell). Fighters can do their thing encounter after encounter and have a lot more control and agency in the game relative to the magic user.
In 5E, Wizards are empowered much further in this regard. It's a fundamental issue with the 5E game design. Wizards (and other spellcasters) are given too many tools to affect play compared to other non-caster classes.
I know a lot of people complain that magic users only get so few spells. Personally, I'm ok with it, I see the reaction to such complaint has been to push the balance too far in the other direction.
To fix this in 5E... I'd do the following.
1. Remove at-will cantrips from the game.
2. Keep the number of spells per day - but maybe cut down across the board with extra magic abilities for spell-casters.
3. Allow spell casting to be interrupted - if you are hit before your spell is cast you lose the spell (or require a Concentration save to keep the casting).
I'd look at how well classic D&D handles the balance between fighters and magic-users, and I'd implement rules to bring 5E closer in line.