At this point I don't even know what you're talking about. Where did this idea of all casters come from? And even if your society was all casters how is a d4 Wizard any weaker than a d4 commoner?
Look, the idea is very simple. Take two equal tribes/cities/kingdoms/whatever. One is a book standard D&D society with standard class and wealth distributions and by the book access to magic items. The other is the same size but has no magic. No Bards, No Clerics, No Druids, No Paladins, No Monks, No Rangers, No Wizards, No Sorcerers, No Shadowdancers, nothing with spell casting or a (sp) or (su) class ability. Also no magic items. No +1 swords, No potions, No wands of CLW.
When these two societies come into conflict, how can the 'muggle' society survive without some mechanical advantage to offset the use of magic. No superior terrain, or numbers either, those are mechanical advantages.
We're saying the muggles are doomed, You're argueing in strange directions that have little to do with the discussion.