For me? DMing became difficult at harder levels. I'll allow that is just as possible (if not probable) that it was my problem and not the game's. I also began to dislike the arms race and the preponderance of magic, magic, magic.
For my players? The excessive splat books which - again - could be a problem attributed to me (the DM) as opposed to the game. Otherwise, I think it's fair to state the players really enjoyed 3e (as did I, when I was a player).
You won't hear me (I hope) bad-mouthing 3e. I think it's rude, to be blunt, to heap negativity on another guy's choice of system. 3e brought me back into D&D, as it did with a couple of my friends. I got nearly a decade of awesome gaming out of it. I owe Monte & Co. quite a bit.
To your point (and you of all people know I ramble before I finally get to the point): I don't think we could ever go back to 3e because 4e exposed some of the "flaws" of that game that we, perhaps, didn't even realize were flaws until we played 4e. I know that doesn't make much sense on the surface, but it's true nonetheless.
WP
The reason I ask is that if 5E is to be the "holy grail of D&D", I think we need to look back on both why people love and dislike 3E and 4E.
To your point about splats, I think this is a seldom-talked about, but very important point. If one looks back at 1E, there are really only 5 core books for players with rules - the Player's Handbook, Unearthed Arcana, Oriental Adventures, Dungeoneer's Survival Guide, and Wilderness Survival Guide. That's it. We had 12+ years of an edition with only FIVE player-focused rulebooks. Compare that with 3.5, which had (not including compendia)
24 IN A FOUR YEAR PERIOD! That's way, way too many core rules options for a DM to master. Therefore, the game is bound to break as written (especially in a homebrew) as there is no way for a DM or adventure writer to account for everything. 2E started to break when kits took off, and 4E shows no signs of curtailing the problem as there will be 5 Core Player Rule Books within 1 year (Player's Handbook, Adventurer's Vault, Martial Power, Arcane Power, and Player's Handbook 2).
I'm all for options, not restrictions, but the drive to sell player-oriented books (in both 3E and 4E) is, IMO, killing this game. It's too much. I wish D&D were an evergreen, one-shot type base product, that says "here are the rules for players in this one book (or maybe a small few spread out over time - such as 1/year - but not 24!)" and the rest is using those rules (adventures, campaign setting, player and DM aides).
The other problem I have with 3E and 4E is the time it takes to play the game. I have mentioned this elsewhere, but I think 5E needs to be released as a game that is played without the grid & minis. That type of play can be added back in a tactical sourcebook, but otherwise I think it has the effect of making the basic game too centered on combat simulation and not on adventure (which I define as seeing unique places and meeting various challenges). By "too centered" I mean that a single combat takes up too long out of a session.
Of course, the economic realities at WotC may prevent such a strategy, but I think that for the health of the game, it needs a format change.