Just a few thoughts off the top of my head... it's 1 AM and I really should be going to bed.
First things first: Chris, I forgot to check in on that Nutkinland thread. Regarding your question there regarding RPGs and wargames, you hit the nail on the head. Wargames are so far in the past that referencing them does no good other than to incite an emotional reaction in people.
Anyway, on to the matter at hand.
I think the sticky thing in RPG publishing is how we define quality. I have a few theories that I've had cooking in my head for a while. I'm going to run them up the EN World flagpole and see if anyone salutes them.
RPG material is hard to categorize by quality based on a browse in a store. A game or supplement's true value doesn't really shine through until you use it in play. Aside from obvious howlers, most stuff works in rather subtle tones. How many people realized how important Spot and Listen were until after they played a few sessions of D&D 3? I've seen people criticize EMPIRE for including example unit stats for the mass combat system, but what would those people think if they needed to run a mass battle on short notice in their campaign? Suddenly those "pages of wasted space" are incredibly useful. Even if a DM has a lot of lead time, the sample units can save him 30 minutes to an hour. Sure, they're not the most interesting thing to read, but wouldn't you rather have some extra time to prep an adventure or relax?
So, I think it's hard to read an RPG book, never mind flip through it, and get a complete picture of its value.
What I think happens with RPG purchases is that consumers make a pretty basic, gut-level assessment of a book's utility. Based on that, they make a purchase or pass it by. Look at the Quintessential Rogue. I hear a lot of people complain that they don't like the Quints. But can you think of a single book that so simply and so directly says "Hey, this book has all sorts of rogue stuff in it!" It passes the flip through test with flying colors.
Here's a challenge to you all: look at a book's title, jot down what you expect to find in that book based on the title alone, then look at its table of contents. You'd be surprised how often books fail to match your expectations. Even more interesting, a lot of the time you can't really begin to list what the book holds. I think this an important test for any book's viability.
Here's another point: I think a lot of my success as a designer stems from my realization that what I know about and want from RPGs doesn't have much to do with what the average gamer wants. I'm NOT the average gamer. I've been playing for over 20 years, I've read and owned hundreds of gaming books, and I'm a walking library of obscure gaming knowledge. The guy who used to play D&D in high school, has just hit 30, and picked up a PHB for nostalgia's sake has nothing in common with me. Gaming with non-hardcore people is an incredibly useful way to learn how people play D&D. I've learned more about design watching 16 year olds play through RPGA tournaments than anywhere else. This feeds back into my earlier comment that quality is hard to pin down. What I think of as innovative or cool doesn't necessarily match up with what they see as useful.
I also think there's a major legacy problem with RPGs. I think a lot of gamers have been burnt by purchases that looked cool but didn't work so well in play. They're nervous about getting stuck with another lemon and don't buy anything unless it has obvious utility. AD&D2 had such crappy mechanics that bad rules are still considered the norm for a lot of veteran gamers. They expect stuff to be broken, so when it is they don't really think to complain. Gamers are remarkably forgiving when it comes to messed up mechanics. This makes that initial impression of a book even more important, since in some cases the subsequent value a book has either fails to have any effect on a gamer's buying patterns or drives him to become a bit gun shy about purchases.
I think Mutants & Masterminds is an excellent example of a book that hooks prospective buyers in with the cool factor then keeps them with exceptional mechanics. I love M&M, but when it was first announced I was pretty cool towards it. I'm not a huge comics fan, and I've never really run a supers game.
Then I saw the book, and oh man did I want to run it. The graphic presentation was so gorgeous, the art so dynamic and interesting, that I felt a primal urge to RUN THIS GAME. Heck, I showed the game to 2 of my friends and they both marched out and bought it after flipping through it. The book is just too cool for words.
When we sat down to play it, I was hooked. My friend Jared was playing a speedster who could generate fire. At one point, his character was battling a berserk robot on a city street. His fire blasts weren't doing much damage, so he decided that he wanted to melt the asphalt the robot stood on to trap it in place. That's not a use of his power covered directly in the rules, but it was a piece of cake to resolve without any real improvisation. I simply assigned a DC and had him make a power check. Voila! He succeeded, and the robot had to waste an action busting free. That was the second time (the first being my initial session of D&D3) that a system gave me the mechanics to handle a completely oddball situation. The rules didn't explicitly handle it, but they were easy to extend to cover it.
To this day, I buy M&M books when they come out, I still flip through the game, and I'm dying to run it again. The game was like a virus: the friends of mine who have seen and played it have all bought the book.
So, in summary I think that the problem facing the d20 market is that we still haven't learned what questions we need to ask. d20 has caused a lot of fundamental changes in this business, and I think some of these changes still haven't come to light. I mentioned in an industry forum that we're in the third generation of RPG design. For the first time ever, we have the ability to listen to gamers and react to what they say. I think d20 is a major reason behind this development. The more we start thinking like gamers who buy maybe one book a month, play once a week, and see D&D as a game to be played (as opposed to theorized about) the happier we'll be.
Whew! I hope that's coherent. I didn't mean to ramble quite so much.
