I've never liked the boxed set concept. A typical justification for it would be that it is a single large system book split into two player and GM focused books with an accompanying starter module. In practice, you are probably better off selling all three separately in traditional formats and not passing packaging costs onto the customer, and if you must sell them together simply shrink wrapping the three together would be better. However, both the boxed set and the shrink wrap concept have the problem that I will never ever buy a game book that I can't browse through first.
The pdf has the same problem. I'm not going to buy the pdf unless I can read it and determine it is quality work that will see use at my table, but of course the pdf seller can't really allow me to have the whole pdf until I pay for it. The result is something of a stalemate that generally prevents me from spending much on pdfs - especially since 90% of the pdfs I've bought I regret.
Any game that requires a bunch of fiddly props and pieces to play will IMO interfere with imagination. If your game requires lots of custom dice, weird tracks and boards, spinners, fortune mechanics, and so forth, you should probably think about designing board games instead of roleplaying games because you've entirely missed the central aesthetic of play. So if you have a boxed set to contain those, that's bad too.
There is however one concept that I think is well suited to the box set, and that's the adventure path reprint. Paizo's reprints of classic APs in hard back format would have been IMO better off as box sets. That's because the heart of any good module IMO is good looking maps, and maps can't really be printed in a hardback. Putting your maps in a big thick book is a cardinal sin.
Game books IMO have to be large full folio prints. Any book printed in a smaller format would be better off half as thick and twice as big. I loath the teeny little 2e handbooks.
In practice, this seems to provide for roughly 3 formats, all of which were hit upon by TSR fairly early on. First, the core book which needs to be IMO about 300 pages and be as sturdy as you can make it and hit the price point. If you have a choice between lots of color and gloss and sturdy, go with sturdy. A lot of the 3e hard backs in my opinion hurt themselves because of the notion that they had to have lots of full color artwork, and the problem is that bad artwork is worse than no artwork. If you have a choice between lots of artwork, and a few iconic pieces by good artists, go with the few iconic pieces. And black and white ink comic book style art is IMO still classic and often cleaner than paintings, especially for small illustrations, if you can find a inker that is good at it. In fact, in general, go for comic book art whether black and white or color to stay out of the uncanny valley except in the cases of a few real masters of the craft. Most D&D artwork since 2e has been real crap. 3e and 4e were overall terrible, though 5e seems to be going back in the right direction they are still emphasizing 'more is better'.
The only place more is better is a MM, because any monster that doesn't bring to mind an archetypal image is not inspiring. You don't really need art for a vampire, but you better have art for anything less a part of the common imagination. But otherwise, the same sort of rules apply. Good black and white inked comic book art is better than bad paintings.
The only real innovation I would see here is that if you can find someway to have a CD with a pdf of the work with selectable text glued into the back cover, it would be a huge bonus. That said, it may make more sense from a price point perspective to just sell the pdf separately, though I for one would love if there was some sort of coupon or redeemable code for a discount on the pdf you could get with the purchase of the physical work. Not sure how that would work logistically though, unless you were selling the physical book through RPGNow. Obviously, the real world brick and mortars would be pretty diverse, and if you had the coupon sealed in the book you might as well just include the CD instead.
The second format is the module, which generally for modern rule sets and content standards should be 64 pages of dense text. Art should be minimal, and if you are blowing your art budget on something other than maps and covers you have never been a GM. Since a module is likely to be the only thing you'll get someone to buy shrink wrapped, the cover has to sell it. Monster stat blocks should be in the encounters that they refer too. If you have to flip back and forth in the module to find things then its failure. To the extent art should appear at all, it should be entirely illustrative, such as the handout in the original Tomb of Horrors. The only person going to see the book is the GM, and he doesn't need illustrations of someone else's party doing stuff in the adventure. In fact, he rarely needs illustrations at all. Encounter blocks should have short well written descriptive text separated from the rest of the encounter as pioneered by Tracy Hickman. In fact, I6 Ravenloft should pretty much be your template for module design. The map should be large and gorgeous. It should be separate in some fashion from the text, either as a detachable cover or an included folded poster.
You could use the module format for small supplements, but I find these tend to be a bit too small. That has for me tended to work only for third party supplements where the seller is filling in a hole in the system rules than the designer should have never had in the first place.
The third is the small supplement that doesn't quite arise to being a hardback. A good size is 96-128 folio pages perfect bound. I think that 3e really made a mistake in trying to make every supplement a hardback. It typically meant that each book was bloated with filler, had too wide of a focus, and for me had a price point I couldn't justify from the percentage of the book I would actually use. More smaller supplements probably would have lured me into more purchases. For supplements of this side, you really need to be asking, "Is this a primarily GM book or a player book?" If it's trying to be both, you probably are creating a Bard - jack of all trades, master of none. That raises up another problem with 3e's all hardcover format - it made things that had been traditional part of the province of the GM seem the province of the player, contributing in my opinion to the Christmas tree/optimization focus of late 3.5. Basically, by combining GM and player books, you were encouraging players to impose on the GM's setting without realizing they were doing so. You could also use this format for larger adventure. Again, I think having the maps shrink wrapped together with the book is a better format than a boxed set, and the perfect format here is probably foldable card stock similar to a DM screen where one fold can be used as a back blurb for the set.
For supplemental stuff it's ok (plain text PDF for searching, character management as software), but not for reading the rules.
