The design I went with for my homebrew system to try to balance "cool magic" and "one-shotting a PC or BBEG sucks" was to implement saves.
A save was an ability that could be used once per encounter to downgrade a crit to a hit, a hit to a miss, or a miss to a mishap. I was using a PF2-style four-level success system, where people had passive defense scores, and the attacker rolled, even for spells and traps and such.
There were a variety of saves, each of which would both give you defense and provide some boost on your next round, so that you had an incentive to wait to use them at the opportune moment. For instance, the Basic Reflex save would downgrade an attack, then let you make a combat maneuver against your attacker (grab, trip, shove, disarm, etc). The Basic Will save would downgrade an attack and then grant you a round of immunity from mind-affecting effects, giving you a moment of lucidity.
Each PC would get saves from their class or feats or magic items or whatever, but you could only have a limited number primed at a time. Whenever you finished a short rest, you'd choose which saves you had ready going forward.
The bonus effect was that higher-level martial saves created nice cinematic moments where the hero parried a perilous attack and turned the tide. There was lots of "downgrade an attack and force your attacker to move 10 feet, with you following" which was the first time I got that lovely Princess Bride "chatty duelists"-style movement across a battlefield.