Reviews you can trust?

[Aberzanzorax]

I've begun to see a pattern of 5/5 star reviews with almost no counterbalance of 1/5 or even 3/5 in most places I look (including here).

While I'd hope to see a bell curve of scores on the one to five range, it seems most products get a 4, with the remainder getting 5s or 3s, and very few getting 2 or 1.


Do others see this? Are there places for rpg reviews that are more balanced (or if not sites, reviewers)? Are my perceptions off?


I guess, what I'm mostly hoping for, is not a "review" as an "I'm excited about this book, so I'm going to review it, because I liked it so much, and I can gush about it" but rather as a critical analysis, examining high points and flaws and giving, ideally, a summary of the quality as well as a fair score, with average really being an average score.


What have you all found?

 

[Crothian]

I've basically given up on reviews. I think most of the good reviewers have stopped and it became too much like work to try to find one that actually gave information about the book.

 

[billd91]

I've begun to see a pattern of 5/5 star reviews with almost no counterbalance of 1/5 or even 3/5 in most places I look (including here).

While I'd hope to see a bell curve of scores on the one to five range, it seems most products get a 4, with the remainder getting 5s or 3s, and very few getting 2 or 1.


Do others see this? Are there places for rpg reviews that are more balanced (or if not sites, reviewers)? Are my perceptions off?

I don't think your perceptions are off. But you have to realize the reason. There's a ton of inherent bias in the selection of materials to review. Without someone assigning out everything for review to a body of reviewers in some way shape or form, you'll only get reviews of materials people read and then feel a need to tell people about. And the reason people feel the need to tell people about it is either because they want to give a positive review and recommend it... or warn people away. So you're generally not going to see a whole lot of middle-of-the-road reviews compared to higher ratings or very low ratings. And to add to that, you'll probably see generally fewer low ratings because those reviewers were probably at least a little careful when deciding what they were going to buy in the first place, weeding out many of the products they thought they wouldn't like in the first place.

Let's face it, the RPG supplement industry just isn't the movie industry. It can't really sustain a side industry of product reviews (and that's assuming that the movie industry can really do so given the struggles Ebert's latest review show has in gaining funding).

 

[Morrus]

Yeah, I think the issue is that most stuff that people buy and read all the way through is stuff they like. The stuff they don't like either doesn't make it onto their shelves, or - even if purchased - doesn't get read.

Nobody has a full-time reviewing job where they're assigned things to read whether they like them or not. The movie industry supports such professional critics, but not the RPG industry. It's too small.

 

[Morrus]

Yeah, I think the issue is that most stuff that people buy and read all the way through is stuff they like. The stuff they don't like either doesn't make it onto their shelves, or - even if purchased - doesn't get read.

Nobody has a full-time reviewing job where they're assigned things to read whether they like them or not. The movie industry supports such professional critics, but not the RPG industry. It's too small.

 

[saskganesh]

Gaming reviews, whether for tabletop, online, or computer are generally poor. It's probably worse than the music industry.

Some of the factors include:
1) The reviewer lacks a critical framework. They don't know enough about enough games to make a good evaluation beyond "it's shiny and NEW and it's fun for me."
2) The reviewer is a fan already. This isn't always a problem, but usually it means flaws are glossed over.
and
3) Lack of editorial support and general unprofessionalism. A website or magazine is usually beholden to advertisers, who publish games. A bad review threatens that relationship, so the temptation is to retract the bad review and throw the reviewer under the bus. Since the reviewer is probably a fan and unpaid he or she is probably easily replaceable.

All is not lost though if you are looking for good information. I think fan/user comments on media like metacritic and amazon give a very good picture of the value of a product, as well as threads on message boards such as this. 'Word of Mouth" or "Word of Keyboard" remain the best sources of information about a new game or gaming product.

The only problem there is one might have to read a whole lot more.

 

[enrious]

I've also basically given up on them as well because the people who I remember giving great reviews here don't do so any longer and over at my other stomping grounds, a reviewer essentially does nothing more than recite the facts of the product (page count, margins, etc.) and then finishes up with a paragraph or two stating that they like it. And they're considered one of the most popular reviewers! (To be fair, that apparently surprises the reviewer as well)

So apart from reviews that lack substance, there are reviews that make you wonder if the reviewer even read the product. As an example, there was a review that came out over the summer that praised a new product a week or so after its release, which surprised me because by that point there was already a thread nine pages long that pointed out the errata and mechanical flaws (which are fundamental) - the product got a value rating of 4.5, despite being given a content rating of 3.75. Are you kidding me? Rightly or wrongly, I've dismissed the reviewer because there's no way I could trust their reviews - either they just don't understand the mechanics when they read the book, skimmed through the book, or didn't want to say anything bad because it could jeopardize future free swag.

Maybe I was spoiled by some of the regular reviewers from 5-10 years ago. Maybe not. These days, I've found the best thing to do in order to get a better handle on a product is to start a thread about it and get a consensus.

 

[AeroDm]

The problem with reviews is, in my opinion, two fold. First is that there is no mechanism to motivate a reviewer to be critical or careful with their opinion. It isn't like professional reviewers where their reputation follows them around and has to be protected. Second, most people who review *want* their review to be read. Sometimes they review as a form of promotion for their blog or something, but often it is also just the way they've decided to participate in the community. As a result, the main time you see a review is when the publisher of a product promotes a good review. Since a publisher is not motivated to promote bad reviews, reviewers are motivated to give good reviews.

One solution I've seen comes from beer review forums. They include not only the review, but a standard deviation of the review from the reviewer's average. This way, if someone with an average of 4.8 gives out another 5 star review, everyone sees it as "5 star (+0.2 average)" and the raving enthusiasm is moderated by the reality that this guy only gives out really high reviews.

The downside to that, though, is that when the reviewer eventually wants to "save" their reputation and come across a product they sort of dislike, they are now motivated to start throwing 1 stars around to balance it out.

 

[Aberzanzorax]

Interesting responses, all.

I'll agree with others that reviews seemed of higher quality 5-10 years ago. I wonder if that was because the market was "bigger" or less fragmented?

I'm thinking specifically of the d20 glut (and perhaps the "warning away" as much as the "enthusiastic sharing") and enworld reviews.

I found it very, very useful to read reviews here during the time period of, say 2002-2007.

I wonder if the decline in quality of reviews is due to the edition split, and the edition war (someone posting positive reviews because they like a system moreso than a product?).


I also wonder if perhaps reviews are simply less needed now than before? During that time there were so many products, there was enough of a need for assistance in determining which were poor and which were great. Now it seems a small enough pool that I can do my own research without as great a need for advice.

Similarly, I wonder if number of products allowed for more of a "baseline" to be determined, whereas there isn't a clear point of middle of the ground now?

 

[Morrus]

I'll agree with others that reviews seemed of higher quality 5-10 years ago. I wonder if that was because the market was "bigger" or less fragmented?

There's probably an element of rose-tinted-nostalgia going on there. I personally don't remember reviews being any better.

There were a lot of really bad reviews back then.