The history and reasoning behind the D&D (Basic, Expert, RC, etc.) vs. AD&D split, as I understand it:
In 1974 there was Dungeons & Dragons, 3 booklets in a box, written by Gary Gygax and Dave Arneson. This game was very loose and freewheeling (though whether this was intentional or not is debatable -- certainly all the campaigns in Lake Geneva all remained pretty close to the baseline that later became AD&D). Over the next 2 years several supplements and magazine articles added many new rules and complications and keeping track of it all became unwieldy (plus the game was starting to become popular with a non-hobbyist audience who needed/wanted more explicit rules and instruction than were provided by the minimalist original rules).
Thus, in 1977, it was decided to do two revisions of the game. The first, Basic D&D (Gygax/Arneson, edited by J. Eric Holmes), would serve as an introductory set for the non-hobbyist audience. It would include a stripped down/simplified version of the rules (only covering character levels 1-3, leaving out many of the more complicated options and rules) along with a more straightforward and simpler to understand explanation of how to actually 'play' the game. The second was Advanced D&D (Gygax as sole author), a series of expensive hardbound volumes that collected all of the best material from D&D + supplements + magazines and integrated it into a single cohesive (at least comparatively) whole. Advanced D&D was D&D the way the folks in Lake Geneva played it and felt it was best played. But, realizing that not everyone wanted a single uniform ruleset and some people prefered the looser, more freeform style of original D&D, it was decided that the original rules would also be kept in print (though no longer supported with new supplements). Thus, in 1977-79 you could buy the D&D Basic Set (boxed set with basic rulebook, dice (later chits), and introductory module), the Advanced D&D hardbacks (Players Handbook, Monster Manual, Dungeon Masters Guide), or the original Dungeons & Dragons boxed set (white-box with 3 booklets and 'Original Collector's Edition' blurb on the cover). One (presumably unintentional) point of confusion is that while the Basic rulebook consistently refers readers to Advanced D&D for coverage of more advanced/detailed rules (characters above 3rd level, additional character classes, etc.), since it was written/edited before Advanced D&D was actually finished (the Basic Set was released a couple months before the Monster Manual, about a year before the Players Handbook, and almost 2 years before the Dungeon Masters Guide) its rules are actually much closer to original D&D than they are to AD&D.
So far this is all pretty straightforward. Where things begin to get a little more confusing and complicated is in 1980 when TSR decided to finally take the original white-box D&D set out of print and to revise the Basic Set. For whatever reason (and this is where the rumors of Arneson's threatened legal action come in -- that something called D&D and crediting Arneson as co-author must remain in print), instead of making the Basic Set more compatible with the now-complete AD&D line (by, for instance, having the AC scale start at 10 instead of 9, giving clerics a spell at 1st level, giving fighters d10 hit dice, etc.) it was decided to keep the revised Basic Set's (Gygax/Arneson, edited by Tom Moldvay) rules essentially the same as the 1977 version's (and thus more compatible with original D&D than Advanced D&D). And instead of refering readers to Advanced D&D for higher level play, it would refer them to the newly-written Expert Set (Gygax/Arneson, edited by David Cook with Steve Marsh), a companion volume that consisted, essentially, of a revised version of the original D&D rules (there's almost nothing in the D&D Expert Set that doesn't come straight out of the original white-box rules). Thus, starting in 1981 TSR had two parallel in-print and actively supported D&D rulesets -- AD&D as the hobbyist game, the game for 'serious' players, and Basic/Expert D&D as the 'popular' mass-market game targeted at beginners and more casual players. The idea of a ruleset to appeal to those who prefered the more freeform style of original D&D had seemingly been dropped, as Expert Set D&D (at least IMO) didn't feel particularly more freeform than AD&D, just simpler (fewer character classes, fewer spells, magic items, and monsters, less detailed and complicated rules). As the modules and Dragon magazine articles of the era show, AD&D, not Basic/Expert D&D, was where most of the 'experimentation' in rules and style was taking place -- Dragon magazine was constantly presenting new classes and optional rules for AD&D, and modules like Expedition to the Barrier Peaks (sci-fi), Ravenloft (gothic horror), and Dragonlance (epic romance) were stretching the 'AD&D genre' well beyond its original 'swords & sorcery' baseline, whereas Basic/Expert D&D received comparatively little attention and thus remained largely self-consistent (note that, for instance, while AD&D modules from 1982 on were all set in different not-necessarily-compatible 'worlds' -- some Greyhawk, some Krynn, some Ravenloft, and eventually Oriental Adventures, the Forgotten Realms, and more -- ALL of the D&D modules and supplements from the beginning to end of the product line were set within the same 'Known World' setting).
Starting in 1983-84 is where it gets REALLY crazy and I'm convinced TSR no longer knew what they were doing (non-coincidentally, this is the same time as the management changes occurred which almost drove the company into the ground in 1985 and which led directly to the second management change which DID drive the company into the ground in the 90s). With the revision of the Basic and Expert Sets in 1983 (Gygax/Arneson, edited by Frank Mentzer) and the expansion of the line with the Companion Set (1984, Mentzer), Master Set (1985, Mentzer), Immortals Set (1986, Mentzer), and the Gazetteer line of supplements (1987+, various authors) D&D was no longer merely the introductory version of the game it had been in 1977 or the mass-market-friendly version it had been in 1981, but was now a full-fledged rival product line to AD&D (which was essentially drifting by this point, having lost the guiding vision of its creator, Gary Gygax, who was forced out of TSR in 1986), every bit its equal in depth and complexity.
Things really reached a head with the release of 2nd edition AD&D in 1989 (author/editor: David 'Zeb' Cook), which attempted to both simplify and 'genericize' the AD&D game (i.e. moving it closer to what the D&D line had originally been) and the release of the ]D&D Rules Cyclopedia in 1991 (author/editor: Aaron Allston), which combined all of the advanced rules and options from the boxed sets + gazetteers into a single hardback book no longer intended as an introductory product for beginners (i.e. moving it closer to what AD&D had originally been). Both of these rulesets and product lines were thus serving the exact same purpose, chasing the same audience, and directly competing against each other, despite being published by the same company. It's no wonder both couldn't survive, and it's also no wonder considering AD&D's much higher profile within the company and marketplace (D&D was still stigmatized as the 'kiddie' mass-market version of the game, even though that was longer true, and hadn't been really since the release of the Companion Set in 1984), that AD&D 2E came out on top even though, in retrospect, many people have come to consider the Rules Cyclopedia to have been the superior ruleset.