Mr. Patient said:
I agree that the star system is not terribly useful. I think it needs to be finer-grained (half-stars, perhaps), and there is indeed quite a bit of grade inflation (which I may be guilty of myself), particularly for the older reviews. Three-star reviews really ought to be a lot more prevalent, so that the truly exceptional stuff is more visible.
When I write reviews for Amazon and other sites, this is always my assumption:
5 stars: flawless work. The Shakespeare of its genre. The gold standard everyone else should strive for
4 stars: Exceptional work. Worthy of a spot on the bookshelf.
3 stars: A solid product. Does what it needs to do.
2 stars: A flawed product. Fails in its execution of its idea. Possibly some redeemable material.
1 star: An incoherent mess of problems that makes it impossible to tell what the product is suppose to be about. Useless to the extreme.
I've added addendums to Amazon reviews saying things like "I would have given this 3 1/2 stars if that was possible."
I think the real issue is that people equate 3=average=bad. Which is ridiculous, because average is, well, average. Its a solid product that does its job. I totally accept that not everything I produce is going to be this industry-revolutionizing masterpiece that will change the course of gaming (and that ultimately is what the 5 stars should be reserved for). I'm providing short tools to help make the GMs life easier.
So I agree that many reviews are inflated, making more honest reviews appear "bad". There is never going to be a lot that can be done about it, however, because it is the nature of the beast. We can only hope that the average consumer is smart enough to read between the lines.