I suspect that if I ask you what is the goal in D&D you would answer: "gain XP and treasure". If I was asked the same questions I would answer "have good roleplaying experience and immerse myself in the role of a character". Since these goals are so different in playing D&D, we might as well have been playing different games altogether, and there can be no real discussion between us. BTW, if I was to play a competitive game that its goal was gaining experience and treasure I would have played WOW or something of the kind, not D&D.
Gaining XP and treasure is roleplaying. It is the
Object of the Game in D&D. That is in the books or at least it was.
And in all honesty, World of Warcraft is D&D or at least a version of it. WoW is a version of D&D on a computer. That there is a billion dollar industry with over 10 million players roleplaying on computers right now demonstrates how awesome D&D and RPGs are. However, if you are one of the those who don't think World of Warcraft and all the other computer games derived from D&D are roleplaying, then neither should D&D be qualified as roleplaying. In my opinion, WoW, LOTRO, and Warhammer Online are all great roleplaying games. I know some don't believe WoW is an RPG because they have difficulty immersing themselves in the role. I've suggested to them playing a tabletop version of WoW, like D&D for instance, as face to face roleplaying adds more immediacy. And you don't have to wait out grinding times or travel times because 1000s of players are using the same timeline. There are plenty of reasons to play RPGs like WoW face to face rather than online.
I think our disagreement lies somewhere in seeing D&D as something other than a simulation strategy game with a little bit of human assistance to run it. I think simulations work better because of DMs though, so I prefer tabletop roleplaying to computer roleplaying. From what you say, it sounds like you want more of a theatre game with everyone acting in character versus trying to figure out what to do next... which is most of what people do in D&D. I think D&D is the better game, but that is only an opinion. I happen to think that this billion dollar industry exists because it satisfies its customers.
Actually, playing an RPG with mismatched characters is the equivalent of "blocking" in improv theater. If we both have a goal of "kicking butt and looking cool" and the PC you built is MUCH better, I will fail at reaching the goal, you will not. If both PCs are balanced, then we both can kick butt and look cool.
If you play a 1st level fighter and I play a 30th level fighter in 4ED&D, I'm blocking you from success? I don't believe that is true at all. Your PC traveling with mine will succeed far beyond what he could on his own. It's a benefit to your PC to join up with mine. It's a problem within 4E that the power difference between 1st and 30th is so great that a 30th level PC has virtually no benefit to adventure with a 1st level PC, but I think if you stay within a single tier the game should be workable.
class balance has multiple reasons. Competitive play between players has seldom been a game focus (every intro to RPGing I've ever read has said the opposite) by the designers of most RPGs. At most, an antagonistic style with the GM has been embraced.
Not saying players can't play competitively (in the same way xbox players do on Live). Only that taken literally, the outcome is often PvP and angry players. The result is, a lower probability of fun for players, ceteris paribus.
There has never been a rule about PCs not killing PCs. If you choose to play that way it may be because of expectation, but the game never said otherwise. I stand by my assertion that if you play PvP, you will group together as evidenced in dozens of online RPGs. (of course, many have metagame rules for players who can't handle PvP and don't want to deal with it. But that is a size problem with so many players all playing heroes.)