The amount of time things take was the biggest thing I was afraid of in 3.X. It took me longer than I would have liked to learn the system, particularly the optional combat rules like overrun, sundering, etc. but now that I have it down I can use it fast in combat versus PCs and not be daunted by having to look it up.
Though it might be fun, I can't imagine taking 5 hours for only 20 PC/NPCs to get through just 6 rounds. I think this is another reason I never got into 2E, as I was just a bored player watching two rules lawyers haggle it out on who won initiative. Now I run a game that meets every two weeks on a weeknight for only 3.5 hours, get 1 or 2 combats per game with normally over 10 rounds each with 10+ PCs & NPCs, and the fighting encompasses only half of our time. Our familiarity with the system is probably the reason why it goes as fast as it does.
IMO, 3.5 has been awesome. The availability of classes and PrCs has mushroomed with 3.X, and though it's harder to keep track of it all, I usually only have to worry about the 6 or so players and what they are capable of. I don't really understand why some feel it is limited, such as having only 2 feat paths for fighters, as there is so much in just WotC material to make enough combos to last a lifetime with the core classes alone, especially compared to previous versions.
My biggest challenge now is to streamline my homebrew for ease of reference. I am using my own homebrew world and as the PCs meet new people and have questions, I am needing to flesh out the world with more detail and document it all. Societies, organizations, well-known NPCs, important buildings, national history, local common knowledge, heiarchy, taxes, costs, regular inns, reliable shop keepers, pantheon, city layouts, etc, etc. In a way it's a great motivator to get moving with something at least partially original, but on the other hand it would easy to just give up and go with greyhawk or forgotten realms and let the players read as much as they want. The caveat to that would be the players knowing much more about the game world than the DM, and the players telling the DM who the local regent was and what they would think or say.
I gotta say that not one bit of it is frustrating for me, yet. If it were I think I would find out what it was and toss it out to continue with the enjoyment of not only setting up the game but running it as well. Should the game system itself become a hassle (like 2E was to me) I can see leaving the whole thing behind for something more enjoyable.