No, it's the rules. And I'd say more than just D&D, it's d20 in general that's slower.The biggest slowdown for D&D combat are the players, not the rules.
I'm running a Star Wars game using the story of Dawn of Defiance (it's a SW Saga edition adventure path provided for free on WoTC's site), but the game mechanics of...basically Chaosium's Call of Cthulhu (the old-school version tweaked and house ruled slightly for Star Wars flavor).
My group of 3 hard-core gaming veterans, who had been playing 3.X since it came out, are absolutely floored by how quickly combat in this non-miniature, non-d20 game takes. And none of the "drama" or "detail" has been lost in our encounters.
I suppose it's hackneyed, but it's a refreshing return to the old school (and I type these words sitting next to about 1,000 painted miniatures and dozens of battlemats on my shelf), and for one of our younger players, who has only ever played 3.X/d20, it's a complete eye-opener. He's actually now more disinterested in 4E than I am.
No, D&D 3.X combat is slow. I'll probably play 4E as a sort of miniatures fantasy war game (1-off, massive 5 hour miniature battles), but most likely not as a fantasy RPG.