Then you are not looking at the numbers, or you are actively changing the rules so that the cost of doing the thing most people actually do is no longer trivial. As it stands, few groups do more than 5, maybe 6 combats per long rest on average, most groups take only one short rest per long (or occasionally 2), and most combats are only a few rounds long (maybe 5 for a relatively long combat). This, plus the tedium of actually countering spellcasters meaningfully and the deep unrealism of maintaining constant time pressure all the time that never causes problems from short rests but consistently penalizes long rests, leads to precisely what I described above.
That you have not seen it does not mean it cannot be done. The example I gave above is a Wizard holding their own in damage with only four 3rd level spells. They have four 1st and 2nd, three 4th, two 5th, and one 6th to play with, along with six spell levels from Arcane Recovery (max slot of 5th level.)
"I don't see Wizards solving problems excessively, thus no one does" is not a compelling argument.