The trouble with trying to create a supplement that adds complexity, but not power, is that any sufficiently complex system has loopholes, or interactions that the designers did not foresee.
Even worse, when you start adding multiple supplements (or even modular sub-supplements), the number of interactions between the options in those modules grows exponentially, while playtest time stays more or less constant during the lifecycle of an edition
Indeed. Perfect balance in such a system would be impossible. But then, perfect balance is a mirage anyway; the better target is to go for "reasonable balance", and I don't think that's impossible at all.
The counter-intuitive thing that makes it possible is to build the game
backwards - build the most complex version first, then build the simpler version from that, and sell the simpler version first.
The way to do this:
First, silo "things you can do" from "the numbers you use to do it". If you don't do this, the powergamers will immediately latch on to the strategy of building one-trick ponies - pick one power, then throw everything else at the numbers to make that work. Anyway, I'm going to call "things you can do"
powers, and "the numbers you use"
feats.
In the complex version of the game, then, players just get a budget for powers and a budget for feats, and build their character from those atomic building blocks. (And, when they gain a level, they get bigger budgets.)
Under this model, the game doesn't care whether your character gets a "racial hate" bonus vs orcs and goblins because he's a Dwarf, because he's a Ranger, or because his magic sword was quenched in the blood of giants. You get the bonus, and that's enough.
Perhaps incidentally, Mutants & Masterminds has basically done this already.
To build the simpler versions of the game, then, you start bundling the powers and feats together into various combinations. The "dwarf" bundle combines the "darkvision" and "stonecutting" powers with the "racial hate" feat, for example.
And, because there's a loss of efficiency inherent in using a bundle, you give a price break on the cost. (It's the same idea as a "pre-theatre menu" - pick one starter and one main from this cut down list; no substitutions, additions or alterations allowed.) And, obviously, you get different bundle sizes: themes, races, classes... some of these may include advancement options, some may not.
For the Starter Set, then, you bundle everything up into two piles: race and class (or even race-as-class). Players pick one of each, and get playing - at every level they just add the pre-calculated numbers on and add the pre-selected powers. Nice and simple.
For the Core Rulebook, you get to be a bit more flexible. Perhaps you take the 3e model, and have races as before, give the classes a slightly smaller set of 'goodies', and let the player choose a feat every 3 levels.
And then the Advanced Players Guide can present a more 4e-like class structure: classes get very little, but the player gets to choose feats and powers to customise his character as he goes.
Finally, you may or may not present the final level of abstraction, showing off the feats and powers system in all its glory.
---
Players can then progress through the levels of complexity as they wish. They probably start with the Starter Set, with the big bundles of pre-built stuff. At some point, they're likely to want to customise more... and that's fine - they just switch to the Core Rulebook class progression, and start picking up feats as well. (There's no problem with this, since the SS Fighter bundle is a bit bigger than the CR Fighter bundle, with the cost difference being enough to pay for the additional feats. The character remains legal.)
Then, if and when the character wants more control over his feats, that's fine too - rather than taking the next level of the CR Fighter bundle, he instead switches to the APG Fighter bundle and starts picking feats and powers.
For the next campaign, of course, he doesn't have to go back to the Starter Set for his character - he can just build his character at his preferred level of abstraction, and it will work.
But one crucial thing needs to be noted: the game must explicitly forbid substitutions. No, you can't take a SS Fighter but swap Cleave for Weapon Focus. This is important, because the SS Fighter includes a "price break" because it's not ultra-optimised. If you want to swap things out, you can certainly do so, but you need to start paying the higher costs. You're picking off the full menu now, not the "pre-theatre" options!
---
Expanding such a system then becomes just a matter of adding a bunch of new feats and powers into the game (each of which is hopefully small enough that it can be balanced reasonably easily), and then bundling these up into new races, classes, themes, etc. Whether the books then actually present all the feats and powers, or only present the completed bundles, is perhaps a discussion for another day.