One of the chief reasons the rules get revamped is so that more supplements can be sold.
I'm not actually saying that 3.5e came along so that S&F (etc) would be made obsolete...
No, what is actually happening that the rules develop to a point that the framework of the last set of core rules isn't good enough to hold them anymore, and they must be revised to take into account all the design work that has come before.
oD&D was a very simple system - and somewhat incomplete. d6 hit dice for everything. No initiative system in the original set. Only three classes. Things like that.
With the supplements, it quickly evolved into a different system, and the game of oD&D + supplements was vastly different to just oD&D.
When you publish a new book for oD&D, can you assume that everyone has all the supplements and also uses them? You can't - not with any set of core rules. (And oD&D was also hugely house-ruled).
Thus, AD&D. oD&D cleaned up, combined with the supplements and other material, with the odd bit of material that didn't work (see Eldritch Wizardry and its initiative system) thrown out.
Ten years later, AD&D was in the same state. (Actually, 6-8 years later with the release of Unearthed Arcana!) Once again, clean it up, put it all back together and release it so that everyone is on the same page.
AD&D 2E quickly ran off the rails, in fact. The Complete Books kept adding optional material that just wasn't quite a good fit with the basic rules. (The Complete Priest's Handbook is the best example of that - a woeful effort). Player's Option really confused the issue when it came about. Great ideas, occasionally brilliant execution, mostly problematic execution.
D&D is a game where for all of its history, people have been designing new options for it. There is never been a time when some aspect of the game hasn't been tweaked. (And Gary Gygax was one of the biggest tweakers.)
D&D 3E was necessary because of the mess 2E had become. Player's Option had pointed the way forward, but the structure of 2E didn't allow its expansion any further. (The Proficiency system was the biggest problem, btw - see how that gets used in Player's Option and then compare to the Feat/Skill system of 3E. That's the ancestor of today's system).
And 3.5E? Well, 3E worked well, but it had a lot of rough edges. Any system so big is going to have them. From the problems with haste and harm, to the rocketing DCs of spells, to the deficiencies in the weapon size system exposed by Savage Species, and to the overly complex monster generation system (also exposed by Savage Species, as monster characters were being made more feasible), it got overhauled and the problems ironed out. Consider also the importance of the wilderness adventuring sections in the 3.5e DMG and how they then integrate with Frostburn and Sandstorm...
Is 3.5E "the system" then, and there will be no 4E? Not at all. 3.5E has displayed more problems - rocketing caster levels in relation to Holy Word, the ongoing problems with polymorph, an incomplete weapon sizing system (reach & missile weapons need better definition), and so on.
Then too, there are more fundamental questions to be answered. For instance, how should Diplomacy be handled?
However, I do believe that the future of D&D is in very good hands.
Cheers!