This requires you to go beyond the rules and create additional content to "balance" spellcasters, however. If these things were the default, then they would be presented in the game.Farmers cottages, no. Castles, yes.
Or not, and let Wizards be all that. It’s a simple equation.
Before I continue on this point, I'm ok with countermeasures as long as they make sense in the game world (my immersion, man!). If the game world has magic well known enough by the populace that specific counters to specific spells are a thing I can encounter, that's fine.
But saying "well, of course castles have protections against Leomund's Miniature Fortress" is akin to saying "well, of course Wizard towers have powerful magnetic traps to prevent stabby Fighters".