Based on what I'e read, it sounds like the OP is fixated on the numbers game of DM'ing.
The numbers game, as you call it, is pretty important for getting what you want out of a setting/system. If you don't play it, you end up with a setting/system that doesn't work like you envisioned it.
Granted, being a good rulesmith is not the be all end all of being a good GM and I've known some good GM's that were poor rulesmiths. But hammering out what you want to play with is an important step.
3.x is pretty decent by itself...
I disagree. 3.0 was pretty decent by itself when it was first introduced, but 3.5 and later variations are inherently broken, breakable, and a mess. While 3.0 core is solid up to about 6th or 7th level and stays playable up to around 12th level, 3.5 only manages to get even that far by social contract. In general, you are left with a sweet spot in stock 3.x of around 2nd-4th level, with anything departing from that working mainly by social convention and/or large infusions of DM magic.
By that token, I find it's more of an 'advanced' player's game.
While I agree that 'advanced' players like dials, 'advanced' players IMO don't prefer rules that are unbalanced, top-heavy, and require fine graduations of self-imposed limits on how much they agree to break or not break the game. I'm not particularly a big fan as a player of playing 'mother may I play this character?' games with the GM/social contract, nor am I a big fan of having to review PC's not on the basis of the suitability of the concept to the setting but as a simple matter of play balance. You shouldn't have to negotiate how much system mastery you are allowed to use.
This should solve your "Casters Are OP" issue and power creep issue all at once.
Not really, though it is a decent start. Core only, wizards are really no more powerful than sorcerers, as the number of available spells is quite small so forgoing a few of them imposes a comparatively small penalty. It takes splat books full of spells that solve new problems in new ways to move the wizard firmly a tier above sorcerers. The same is generally true of clerics. But it's not true of Druids as long as 'Natural Spell' is available, as the Druid can then basically be the fighter, cleric, and sorcerer all rolled into one. The main difference between tier 1 and tier 2 isn't how nuclear you can go, but whether you can have an answer to everything. As such, I consider the 3.5 Druid to be the worst offender in the game. It's the only core class I looked at and thought, "I really can't fix this, since the concept implies - 'I can do everything well.'"
Beyond that, there are two main issues with Casters in 3.X and 3.5+ particularly.
First, the DC's of saving throws for spells quickly outstrips the defenses of creatures with poor saves. There are a lot of ways to show this, but the easiest is that the DC of a spellcaster's spells go up at a base rate of 1 every 2 levels, but poor saves go up at a base rate of 1 every 3 levels. This results in a situation where a smart caster in 3.X can easily use 'save or suck' to dominate the action economy and as direct 'I win' buttons because over time foes get less and less likely to pass their saves (this is the main reason 3.5 inflated monster HD and ability scores for a given CR, but of course, this had detrimental side effects not the least of which was inflating the DC of the monsters abilities in the same manner). Contrast this with 1e, where as you leveled up the odds of your 'save or suck' spells succeeding against your targets diminished.
There is an easy fix to this that renders 'save or suck' much more of a crap shoot and therefore returns you to the 1e issue where gambling on a 'save or suck' attack meant a high probability of wasting your own action, and that is simply remove the spell level as a modifier of the DC of the spell. You'll probably also want to also remove the 1/2 HD modifier to the saves of monster's abilities for the same reason (but in reverse). In general, this brings you back to the 1e world where the expectation is that powerful characters/monsters usually make their saving throws.
Secondly, it's far too easy for a 3.X caster to substitute his own abilities for the abilities of something far more capable - trading out apparent weaknesses for strengths and becoming good at everything. There is a reason that the answers to, "I want to optimize my fighter, what should I do?" is, "Play a pure caster." It's far easier to become a melee combat monster as a cleric, druid, or even a wizard than it is do it as a fighter. Often you do this by literally becoming a monster. Sometimes it involves summoning a monster. I always had a good (if painful and cynical) chuckle at 3.5 coming out and issue a 'fix' on spells like Haste and Harm, while at the same time breaking spells like Alter Self and Polymorph Self into little bitty super sharp pieces. To fix this later problem with 3.X casters, you have to individually fix the broken spells. Often it's possible to just take the nonbroken 3.0 version and go with that - compare 3.0 Blasphemy with 3.5 Blasphemy for an example. Sometimes it requires writing custom fixes. You can't call 3.5 core 'pretty decent by itself' until you fix shapechange and calling type spells that let players alter their character's CR dramatically.
Where this perception departs from the OP's perception is that he perceives the problem as being a problem with the casters and so has banned them, whereas I perceive the problem with casters as being a problem with spells. The spells are the class abilities of casters. If some of those class abilities are unbalanced, naturally the class will be unbalanced. But it's quite possible to play a balanced caster by selecting spells that don't let you solve every problem (fireball or magic missile, for example). The trick is narrowing down the utility of spells. At that point, the casters regain their balance and non-casters at least in part cease to live in world where they exist at the sufferance of casters (for example, RAW 3.5, the utility of a fighter of high level is entirely in how much his friend the spellcaster is willing to buff him).
Of course, the other problem with 3.5 in particular is that a lot of broken things for non-casters were put in the rules and justified and defended with the fallacious claim, "Well, casters are already (even more) broken, so what does it matter?", and equally bad, the high end CR monsters were rebalanced on the assumption that they'd be facing casters with powerful 'I win' buttons. So this is hardly the end of your rulesmithing if you are trying to restore sanity.
