My argument was that there are three stages to being a character optimizer. These stages are:
1 - Mechanically Naive: (snip)
2 - System Mastery: (snip)
3 - Optimization Sublimated: (snip)
Folks in stage 2 are the optimizers that everyone seemed to get upset at. Folks in stage 3 are fun in any game.
- - -
Now, what does this have to do with the discussion of "system balance"? IMHO, D&D is nearing the end of stage 2, which has taken us from 3.0e to 4e.
That was a well written post and a good analysis of the theoretical framework behind the direction DnD has taken I think... the drive for players to houserule and the drive for the developers to make new versions (and new games) are both caused by the disconnect you eloquently described at stage 1.
But I think where your theory breaks down is the idea that we can get to stage 3 from stage 2. I think stage 2 is a dead end. In elevating balance to the sacred cow of game design, WOTC has stripped more than houserules out of the game, they stripped out all the different ways people used to play the game, and all the individual nuances which elevated it above say, monopoly. Broken as it was, you used to be able to play DnD as a high fantasy, dark, low fantasy, historical, literary genre, cinematic genre. Everything from Lovecraft to Lord of the Rings, from Sauron to Samurais... The system has now lost most of that flexibility. You are much more limited to a very specific type of (ultra) high-fantasy / high magic game.
If you will forgive a crass political allegory, I'm reminded of a quote by Martin Buber back in the 19th Century, in critique of Karl Marx "One cannot in the nature of things expect a little tree that has been turned into a club to put forth leaves." I think D&D has entered the 'Dictatorship of the Balancariat*" and you will find that it is not, as promised a "transitional phase". It is perpetual (until the collapse).
The other problem I see is that in order to get to stage 2, as a customer of DnD I basically have to learn a whole bunch of what I call "Klingon". DnD was originally based on the literary influences of great writers like Robert E. Howard, Jack Vance, Michael Moorcock, Fritz Leiber, HP Lovecraft (all of whom have adolescent appeal but can also be appreciated on an adult level...) and the remainder was filled in by the completely wide open realm of 'real' Mythology and History. Many of the game mechanics were broken, but quite a few could be intuitively picked up based on what you knew from real life, cinema, history or actual literature (as opposed to fanfic, computer games or manga).
Now days you have to be deeply immersed in gamer-geek or MMORPG subculture to understand even the basic concepts of the game. 4E may be a fine niche game for people who like that particular style, but the elements in 4E don't exist in history or in any literary or cinematic genre I have any interest in. So for me it's a closed system, and one which I don't think it's easy to get to from just being a generic person with an interest in fantasy literature or history but no interest in getting invested in 'geek' culture. That to me is the big difference between the type of people who got into the game in the 70's and 80's and the type of people who get into it now.
I think if you want to have a game that is fun and flexible enough to exist outside of the 'klingon' niche, you fix it directly at stage 1, then the DM and the industry game writers like mousferatu can quit worrying so much about game mechanics and explore the limitless possibilities of making fun adventures that grownups can take seriously, whether they be worlds to history and literature or computer games, manga comic books or whatever it is that turns you on.
G.
* Mind you it's not that I think balance is a bad thing, it's not, it's quite necessary, but it is not the
only thing. Just like a plant needs water. But if it's overdone everything else get's smothered (and your plant dies).