It's not the fluff, it's the system itself. The "3e" ruleset is a different game than D&D. "3.5e" is really d20 Fantasy, 2nd edition. It has nothing to do with "not being worthy" of the name.
To make it more clear: OD&D and AD&D were two separate games, though very closely related. The mechanics and assumptions of past games under the D&D label were pretty close, even in the separate D&D and AD&D game lines.
The d20 system released in 2000 was a wholly new game, a new system, that used the D&D trademark. New mechanics, new goals, etc. I can't speak for others, but when I refer to d20 Fantasy it has nothing to do with a judgment on its quality as a game, just my acknowledgment that it is a different game. I sometimes have to remind myself of this when discussing d20 because since it is a different game, it isn't quite fair to apply the rationales and views associated with (A)D&D. For example, I tried to make clear this line when discussing racial class/level limits in another thread when offering rationales for such limits. The rationales offered don't really apply in d20 since it adopted different rationales, assumptions, etc.
I think this new game distinction becomes very clear when some people talk about how they couldn't stomach "old D&D" and left it to play other games, only to "return" with "3e." That is, they didn't like the mechanics and assumptions of D&D and only "returned" when these were wholly changed by a new system. I sometimes point out that they never returned to D&D, since they are now playing a different game. It sometimes comes across poorly, and I apologize.