Some rambling thoughts: One way to approach this question is to compare DnD with software (MS Windows and Office, specifically).
The greatest virtue of 3E is that it's "DnD", which for many of us, was our first roleplaying experience. DnD is familiar, comfortable. But it also brings along all sorts of baggage: levels, classes, hit points, arcane/divine magic, clerics as healers, experience for killing things, etc. It would be difficult to design a system that is true to the core of DnD, while still avoiding numerous balance and verisimilitude issues. Perhaps most critically, dealing with these issues makes the 3E system more complex than necessary, from a purely game-design standpoint (just look at the mystic theurge) - and all that complexity makes the task of introducing new players more difficult than it should be.
So you have much the same situation as Microsoft: you have to keep the "installed base" happy by providing backwards compatibility, but doing so makes the system unwieldy. Microsoft, however, has the advantage that it's very difficult to escape their products (even Linux is not a realistic alternative for the vast majority of users and companies). At selected points, they've broken free of their legacy software (as has much of 3E), yet it's still Windows in look and feel. It's also still buggy, unsecure, and expensive.
DnD has plenty of competition, but is still currently *the* system of choice. The more unwieldy and complex this system, though, the smaller the number of players. The smaller the number of players, the less economical it is to maintain the system. If more players leave DnD than new players enter, you have a problem. I don't believe DnD is in a "death spiral" as yet, but I can easily visualize 3E drowning in complexity and sacred cows, in much the same way 2E did.
By the way, I see at least one way out of this problem: embrace technology. Who cares how complex character creation is, if I can purchase a program that walks me through it in minutes? Who cares how many combat modifiers there are, if a combat program can perform all the calculations in the fraction of a second, without slowing down play. In many way, despite the advances of the 3E system, DnD feels like DOS - it works, but you have to be a "guru" to figure it all out. Maybe what 3E needs most is the RP version of a GUI.