Your point 2 is the thing that your badly ignoring. If a caster is always behind on damage behind on hp & always behind on ac they need to either reliably be completely indispensable when they step out or they need to have the significant improvement of the situation they bring to the table to last for long enough to have more impact than a couple crits would have.
A high-level caster has enough spell slots to significantly improve the party's effectiveness for the entire adventuring day. You're acting like once Haste ends, that's it, nobody in the party can ever be hasted again. Just...cast it again. Easy.
Look at it from the WoW raid analogy point of view. More than one of your examples of a great buff actually hinder the party & would get the caster added to a do not invite list if done regularly. Out of the few that could improve things the effect is so small that almost any other class is likely to be more valuable to the raid given how far they lag the rest of the time. Hanging your hopes on a highly specific party composition to rescue a boat anchor of a strategy for many other party makeups only works if there are a bunch of other spells that do something amazing for other highly specific party compositions & that frankly is not even slightly the case.
D&D is nothing like WoW in this regard. You don't build your character and then go hang out in a lobby, waiting to get chosen for a raid by persons of unknown composition, hoping to find a party that matches up with you. You build your character along with the rest of your party over months or even years of play. You should be choosing spells that synergize with your allies' abilities, not basing them off some build you found online.