There's a major problem with boxed sets, which is not obvious to the US consumer: Taxes in foreign countries. Specifically, a lot of countries have a value added tax -- a sort of national sales tax (V.A.T. in the UK, G.S.T. in Canada, etc.). It can be hefty, as in 10%-20%. It is often the case that books are *exempt* -- but something in a box will be considered by customs to be a *game,* and get the tax slapped onto it.
For a lot of publishers, non-US markets are very important. I suspect that as they increased in importance, the fact of the different tax treatment helped encourage more US publishers to avoid boxed sets, which were so popular in the '80s. I don't think the VAT alone did it, but on top of the other factors already mentioned -- people can't leaf through it in the store, far more copies of boxes get damaged in transit than books, and greater expenses for manufacturing, assembly, and storage -- it's just another big straw on the ol' camel.