For me, it was rules bloat, but it was front-end rules bloat inherent to the 3E system. Even before all the supplement books came out for 3E, the number of fiddly bits in the game increased exponentially from 2E to 3E, and creating characters, statting up adversaries, making new monsters, and prepping the game became a LOT more work. At first, I loved the new options available- feats, skill ranks, PrCs, etc. But once a few levels went by and we started worrying about feat chains, PrC requirements, skill ranks and synergy bonuses, etc- my group and I realized how easy it was to get bogged down completely with optimazation and the unnecessarily complexities of the system. This also lead to a degree of metagame thinking we didn't like- to be truly effective, you almost have to plan 3 to 6 levels ahead what feats and skills you want to take, before the events that shape the character's development even take place. We never even got into the Complete books or supplements- the rules bloat before then was already nightmarish.
3E also has a strong "rules mastery" component that is a subgame itself- and many of my players have no desire to spend the time necessary to learn how to optimize characters and make them effective in the rules. This is also IME a big hurdle for new players- they don't know the rules, synergies, and feat chains necessary to make a competent character, and they often get frustrated that other characters are much more effective than them, even though they are the same level.
Finally 3E, more than any other game out there, has an escalating arms race me and my group just didn't like. Changing the assumed magic or power level of the game screws with all sorts of other aspects of the game, and for years I tried to make a grittier, lower magic game. Finally, I realized you can't pound a square peg into a round hole, no matter how hard you tried, and I gave up on running 3E/D20.
I agree with the poster above that said that 3E/D20 is a game that is becoming more fun for players, but less fun for DMs. I'll still play the occasional 3E/D20 game, but I won't ever run it again- its not my system of choice by any means.
My time now is much more limited, especially since I have a full-time tenure track teaching and research position at my university now. My group started looking for a system that gave us the ability to customize character abilities while not getting mired down in complexities, had a simple fast-moving system, and that could handle large combats quickly and easily. D20 can't do any of this- its not inherently designed for it. True 20 is great- it goes a long way towards fixing the problems we have, and Warhammer FRP2 is awesome- its very simple but highy customizable as well. But Savage Worlds is our system of choice now- it seems like it was customed-designed for my group. I've run everything from super-heroic to gritty games with SW, and its worked perfectly with minimal bookkeeping and complexity, but with just as many character options as D20. Plus, its a breeze to stat NPCs and creatures out. I've also found as I have fewer rules to worry about, I tend to focus on the plot and making interesting characters and locations- and my love of gaming has returned. For us, a less clunky ruleset really is more fun.