So many, many things
Previous posters have touched on a few...but to summarize:
1) Squandered huge amounts of cash: It may be hard in this day and age to comprehend what a phenamonon D&D was in the early 80's and hence how profitable it was. But it was. The results: many freinds and relatives of the Blume brothers (who had taken control of the company for various reasons) on the payroll with company cars. Questionable investments, including in a needle point bussiness (yes really), and a generally lax attitude towards profitability and the realities of the bussiness that would continue for many years.
2) Alienated their best creators: the big, big, example here is Gary Gygax (our own Col Pladoh), but he is not the only one. If you went through a list of people (positively) associated with the game over the years, you may be surprised by how many stopped being employees of TSR (and later Wizards), and when. On the other hand, mediocre talents seemed to always be able to hang around.
3) Poor execution: TSR had huge successes, including releasing AD&D and B/ED&D at just the right time, the early Dragon magazine, and the D&D cartoon (all basically thanks to Col Pladoh). But for a lot of things, they just made strange decisions and couldn't quite get things right. The dragon dice and book trade fiascos touched on above are some examples. This included all sorts of issues with top managment (see 1) that continued through various owners, and a trend towards brining in non-gamers to "fix things", who of course didn't under tand gaming and made things worse.
And they did make questionable decisions regarding the games they developed and supported. E.g. having many, many similar fantasy worlds, and supporting all of them with multiple releases every month. One of my favorites was continuing to release 1st ed rulesbooks after launching 2nd edition. A source of short term profit, but what about those people who bought those books and then realized they where being phased out...
4) Alienated their customers: Dropping devils, demons, and assasins to apease outside critics, starting a suplement "arms race" that ultimately angered many, many DMs (and not just those with bladesingers in their games). Threatning to shut down D&D fan sites while not having their own web presence (they where on AOL). And just releasing so much crap by the end. (Remember poor execution: there solution to falling sales was to actually churn out more stuff even faster to make up the revenue). You think WotC is taking a flak right now (ok, they are), but for T$R in the final years (as it was called), it was like this month after month after month. Until they went bankrupt.
Want to learn more? The Ryan Dancy link above is good. So are the aniversary products (25th anv boxed set and 30th anv book for D&D), so are the Col Pladoh threads, or just ask, and he'll tell you. And there are many more anecdotes. Why I haven't even mentioned Buck Rogers.