Huh. I was under the impression that there was a previous "death spiral" of sorts with returned fiction. The way I understood it was that Random House started returning books, and were originally repaid with "store credit" (return a thousand books, get a thousand other books). And eventually that got untenable and Random House wanted to get paid for the returns in actual money. I think I got that from some other TSR alumnus (possibly Sean Reynolds who used to be pretty active online), but it's possible they don't have the same insight you do.
That's tended to be overstated, in my experience, but there were some specific problems. The company definitely published too many hardcovers, for example, and they got hit with returns on a couple of those, so much so they ended up remaindering the Buck Rogers hardcover in large numbers. (Remaindering is selling dead stock—dead beause it hasn't sold as expected or because a license to publish the material ends—in bulk at a huge loss, for pennies on the dollar, to recover something/anything for the printed product.) You can still run across dusty copies of that Buck hardcover and a couple other TSR hardcovers in low-turnover deep discount stores now and then. The problem there was overestimating demand for hardcovers for some of the titles/lines but also a function of how long bookstores are/were willing to keep hardcovers around as "new releases" versus paperbacks.
Some of the other fiction lines they started after 1993 weren't getting nearly the numbers the Realms and Dragonlance books were getting, for example, so some returns there would be expected if they were overprinting wildly. But by 1995, their cash flow issues were bad enough they would be unlikely to overprint by insanely high numbers. TSR was carrying so much printer debt they couldn't splurge, especially after they lost the building to the print buyer and were locked in to the printer deals dictated by their new landlord.
The Book Department also let some of the backlist books go out of print after 1994. That was actually a good move for any series releasing a lot of new books every year. You sell out of a print run and decide not to reprint the book. Then you tell bookstores it is out of print and they have a set window of time to return unsold stock if they want. (With series, you can then wait a few years and release a new edition/new printing of the out-of-print book, if demand supports that. In the meantime, stores don't feel like they have to carry so many books in a line.) So there might have been some returns from TSR doing that, but not all that many. The titles declared OOP tended to be books that had sold out at both the TSR level and were largely gone from the RH warehouse pipeline, so you would only be looking at some store returns there. Maybe there were a couple titles B&N or Borders had over-ordered in large numbers, but they tended to be pretty smart about ordering and did not sit on stock they could not sell forever.
At the time TSR stopped paying royalties, which was a couple years before the company was sold, they owed a lot of money to the fiction authors. A lot. Had there been overwhelming waves of returns prior to ending the deal with Random House, they would not have owed royalties like that. Under the TSR contracts, returns are always debited from money owed from sales. But in the end they owed me five figures, even factoring in final returns from Random House. They owed TSR authors with more books in print well into six figures, even factioring in RH returns. The fiction was selling steadily. Not at the rates it was selling in the early 1990s, but steadily.
So there would have been some fiction returns in the middle 90s, but not anything like what happened with the RH returns after the deal was ended.
That book trade death spiral you describe is real, though. It can wipe out companies. The deadliest version is the publisher breaks into the book trade (at last!), has a large number of copies of
Title A ordered, and then a large number of
Title A returned. They have to either pay back money or the distributor wants credit. So, showing the math: The bookstores initially ordered 10,000
Title A, return 8,000 books. You can pay back the money for the 8,000 returned
Title A or you can send them 8,000 copies of your next book,
Title B, provided they even want it. But you have to
print 8,000 copies of
Title B to send them, and that's expensive when, looking at the numbers, you are thinking you are going to sell 2,000 copies, like you did with
Title A. You send 8,000, get 6,500 back. Rinse, repeat with new
Title C. Meanwhile, you are still operating off the payment for the initial 10K in sales and accumulating printer bills for far more than 10K copies when you factor in
Title B (8,000 copies printed),
Title C (6,500 copies printed), etc.
It's possible Random House started using some of the TSR returns to replace money they would have paid out for new orders. Sean could have been referencing that. It would make sense, given TSR's shaky finances and, if that happened, it would have factored into the overall negative math for the company. But I strongly suspect would not have been decisive. From what I saw with my books and from what I know from talking with other fiction authors from the 80s and 90s, the sales were still net positive. WotC paid out a lot of TSR royalty money owed to fiction authors after they bought the company. (Huge credit and thanks to Peter Adkison for doing so.)
If you want a clearer example of the perils of returnability in the book trade, when TSR got so far into debt they stopped printing, the mall stores and book chains ordered heavily from other TTRPG companies for product to fill the now-empty D&D shelf space. And when WotC bought TSR and the product began to flow again, the chains shipped a huge amount of that non-TSR product back in order to clear space for D&D and related fiction. They were not really looking for stock to replace the returns, either, just money back. That effectively wiped out the terrific fiction line at White Wolf and nearly sank the company altogether.