Ignoring the issue of cost, I find myself preferring smaller books these days, both in thickness and in page size. I find the G5 format (169 x 239 mm, slightly smaller than a comic book) used in most Swedish RPGs in the 80s and early 90s fairly optimal for reading. It fits two columns comfortably, and is easy to hold in your hand. Onyx Path Publishing's Trinity Continuum books are sold in a similar format (7x10 inches), and though I haven't actually held them in my hand (only PDFs) I like that size. Ideally you shouldn't go overboard on thickness either – better to find a way to split the book, e.g. a player's and a GM's book, or perhaps rules, spells, and monsters, or whatever seems appropriate.
That said, I'm not sure I prefer that enough to pay a premium for it. A G5 page has about 2/3 the area of A4 or US Letter, so a G5 book would need about 50% more pages to present the same amount of stuff as a regular-sized book, and would thus likely have a higher price point. But then again, that's not a certain thing. Paizo seems to charge about $50 for hardcovers in the 250-300 page range, which is about the same as Onyx Path charges for their 380-page Trinity Aeon book. But it's always hard to compare price points between different companies.