It's unlikely that you will have an aerial army to contend with, so walls will still be greatly effective against an army that has a few fliers.
Your thinking on this is a very conventional and doesn't really take into account the topic at hand and is very depending setting. In Forgotten Realms (which I specifically mentioned btw) flying foes are very likely. If we are talking about Wizards making items,
Potions of Flying would be in very high demand.
Ahghairon's dragonward is one method used in a FR setting to counter flying foes (dragons in specific).
As for passwall, you can make walls 25 feet thick, or just have a plan to let in 25-30% of the army and then dispel the passage, splitting the army and allowing it to be more easily destroyed.
If you let in 25%-30% of an opposing army, you have probably lost the battle. The concept of schwerpunkt especially applies to a fantasy setting. It is why traditional armies try to batter down gates. Making walls thicker is an effective counter and would make them harder to undermine.
Also
passwall isn't the only method of invalidating walls. Also, you seem to assume that it can only be cast once and in only one place.
Transmute Rock can be used to undermine walls;
Wall of Fire can be used to clear defenders from a section of wall and so forth. The biggest limit on most spells is they require concentration. Take a clever group of players and ask them to bypass a cities walls to let an army in and it will happen.
Using teleport, that's a fool's errand in 5e. Between bounded accuracy and the fail chance, you aren't going to get in than a handful of people, and that doesn't invalidate the need for walls or castles. Sure, it could be useful for getting in a spy or a saboteur, but those don't invalidate the need for castles or walls, either.
Bounded accuracy doesn't really apply to the
teleport spell. With magical sensors and/scrying and location becomes Very Familiar. You only need a handful of people to take down the defenses of a castle. A small group is particularly useful for infiltration and opening gates and so forth.
So when it comes to offense vs. defense, it depends on the relative costs involved. Building walls around a city 25' thick is an expensive proposition. How expensive would it be to bypass/undermine them determines their effectiveness. The Forgotten Realms in particular is a very high magic setting with lots of high level characters and organizations. So lots of "unlikely" scenarios are very likely in the Forgotten Realms. Mostly what I imagine keeps things in check is Mutually Assured Destruction. The big events in the Realms take place when one group or another forgets this or there is some sort of worldwide cataclysm (such as the Spellplague).
Using the principle of MAD, it provides an actual use for plausibly deniable assets like adventuring parties to try and tip the balance between large powers/organizations. However, walls by themselves are just expensive speed bumps that can keep primitive hordes at bay for a time.