I'm old, but I'm not that old, so I don't really know where people were buying D&D back in the 1970s. By the early 1980s, AD&D products were available through mainstream retailers.
I remember that AAFES had AD&D in the Garden section of the Ft. Rich PX on a rack; this was before I was playing D&D, so 1979 or 1980, because my dad didn't have PX privs anymore at that point. (He'd gone inactive in the Reserves. He was able to use other facilities because he worked on base, but NOT the PX nor Commissary themselves.)
Switching to boutique stores was a business decision. Anyone know why?
TSR's approach had resulted in more products coming out, but each one being lower sales. Overall, they were spending more but making less profit overall, with rabid fanbases for the various settings.
Retailers want to know they can move product.
GDW went down in 1995, formally closing in 1996 (still in the black), due to a combination of a product entering the book trade instead of the games trade, resulting in having to buy back a lot of unsold copies (the Desert Storm Fact Book, IIRC), and had bought something Gygax wasn't free to sell them (Dangerous Journeys), which resulted in a hefty settlement and taking them out of print. Marc said that they could take a risk and go into debt, or shut down as is and be in the black. they shut down. Marc and Loren were extremely candid with the fans on the mailing list about it.
Palladium was doing fine... in spite of Siembieda's litigiousness. But they were not in many big chains. (I have seen early Rifts cores at either B&N or Waldenbooks.)
Avalon Hill was already almost gone; they were bought by Hasbro at the turn of the century.
SJG was recovering from the Secret Service raid... but wasn't in big box stores anyway... except for
Illuminati and
Car Wars. Both of which were on the shelves of two (maybe more) Anchorage Waldenbooks locations.
This changed with D&D 3rd edition when you could find the three core books at Target and these days you can find 5th edition at Wal-Mart, Target, Amazon (of course), and probably more places I don't show. What changed to make those viable retail outlets when they weren't in 1989? (Amazon didn't exist of course.)
Size of the customer base. Amazon needed to simply be able to sell a case to make it worthwhile. WotC had wormed their way intp Walmart and Target with MTG. MTG was THE hot game earlier that decade... And is still, today, a major factor. There were storefronts in a couple Anchorage Malls that make their rent money selling MTG and Pok-é-mon cards and accessories, from 1996 through 2016 (when I left Anchorage). And one that made their money on lotto scratchers, MTG, and Pok-é-mon... and he was a small shop across the hall of the mall from another card shop operating from a mid-hall booth.
Walmart and Target also have plenty of internal handling, and were able to get enough to have copies on release day. Both have book sections and game sections, and the hype for D&D was a good fit for them... I don't recall which edition Target came on board with (Partly because I've only been in two Target stores... one on the opposite end of the country in 1996, and the other one after I moved to Oregon in 2016...), and for Wal*Mart, I know that 5E was a "special order" item in Anchorage. Keep in mind: there was an anti-RPG Book burning in 2012 in Anchorage at what passes for a mega-church... and the pastor was railing against D&D again according to some of the players at D&D AL...
He apparently had a serious hate for D&D, Palladium (esp. Rifts), Pathfinder, and Ars Magica. (I overheard a student complaining about what said Pastor was railing about, but said student wondered what Ars Magica was about. I pointed them to Atlas' website...)