Adding an enemy caster isn't warping the game, though, and a lot of GMs avoid doing this because they don't like the hassle of tracking them. I agree, which is why I'll usually pick 3 things a caster will do and just jot those down -- they rarely get more than that off in an encounter anyway. So, if occasionally a caster has dispel magic, or a CON save spell, and occasionally smacks the bladesinger, what do you think is going to happen to how the bladesinger engages in melee when a caster is present? That caster might not have any such spell prepped, but the threat remains.
This is very like the 6-8 encounter guideline. If you do that occasionally, then it works even when you only have 2 encounters in a day. The chance that things might go longer will incentivize more conservation of resources in general. You don't have to do it often, but do it.
Now, I agree that if the GM is building an encounter specifically to thwart the bladesinger via special tactics or whatnot, then, yeah, you have issues. But a single enemy spellcaster with counterspell, used on the shield so that the BS is not only hit, but doesn't benefit from the +5 AC for the remainder of the round cycle, will definitely be a moment where PC learn something. And, if no shield is cast, something else will be and that can get the counterspell. Or a dispel magic. If it's thematic for the bad guys, run with it. There's so much that works in lots of situations that also would cause a bladesinger lots of problems, but you need to get comfortable with bad guys that use magic and not just the melee/ranged guys.
EDIT: just to be clear, this is a general you, not a specific one.