Reviews you can trust?

[Gundark]

We all have been bitten by good reviews. Age of worms got glowing reviews, and people recommended to me all over. It wasn't until I ran it I saw it was absolute crap, an opinion that was shared by many who ran the game.

 

[Glyfair]

Also I suspect many reviews are "read-thoughs" rather than "play-throughs" - and there can be a big difference between the two (esp. if you're looking for something to play)!

That would make sense. Most people consider buying a product shortly after it is released. Obviously reviews are most useful to people who are considering buying the product, so reviews based on thorough playtests over a length of time will be of limited use.

 

[gamerprinter]

When I look at reviews for products, I find three problems that make it impossible to trust many reviews.

1) Somewhere along the line, a three-star "average" turned into a three-star "you suck" with five star minimum. I'm not sure when this happened, but it's there. Companies on sites like E-Bay and Amazon get really upset when someone gives them anything less than five star; I've seen reports of people being contacted by companies to find out what they did so terribly wrong that they only got three or four stars.

While I can't account for E-Bay nor Amazon (I never look at their reviews because I never buy products through those platforms), however, looking at RPG review sites and sites like RPGNow or Paizo - I basically disagree with your point of view - on this point, specifically.

While some reviewers use a 10 scale, most of the sites I visit feature reviews based on a 5 scale.

Here is the latest review of my most recently released product, and to my disappointment, it's a 3/5 rated review. If you read the review you will note that Endzeitgeist does not give the implication that this product is a 3 star (you suck) rating, rather it speaks rather highly of many of the aspects of a good publication. His expectations were a bit different than my intentions in the product. Where I didn't want to go 'over the top', End took it as being somewhat bland - a difference of opinion.

As stated above, I take reviews seriously and am looking at how I can improve the product based on points in his review, however, I am hip deep in the development of the next product, Way of the Samurai, which is a similar faction book. As in improvement in the upcoming product versus the yakuza book, we have additional designers working on the class archetypes for more creativity and varying points of view - this should make Samurai a better product than Yakuza.

I plan to make updates to Way of the Yakuza, but that won't happen until after the release of Way of the Samurai, but I will try to fix it based on this review.

Link to the Way of the Yakuza review by Endzeitgeist to prove my point.

 

[Morrus]

Companies on sites like E-Bay and Amazon get really upset when someone gives them anything less than five star; I've seen reports of people being contacted by companies to find out what they did so terribly wrong that they only got three or four stars.

eBay is a special case for two reasons:

1) You're not reviewing the product, you're reviewing the seller. So if someone rates a seller badly because they didn't like the book the seller sold to them, it's the seller who suffers, not the publisher of the book.

2) More importantly, the way eBay is set up makes sellers without perfect 5-star feedback pay more in fees. This is terribly unfair to the sellers, who are better off having no feedback left than having 4-stars. 4-stars, therefore, actually becomes a punishment for them in the system eBay has devised. Unfortunately, buyers don't know this. 5-stars means the seller did what was expected; anything less means the seller did something wrong and gets penalized for it. You naturally think 3-stars would be "this seller did everything OK; average review"; but it's not from eBay's point-of-view. It's "this seller is two points short of complying with the minimum expected of a seller". It's an awful system.

 

[DonTadow]

I'd argue otherwise, [MENTION=22622]DonTadow[/MENTION].

EN World, for example, has moved towards a paid reviewer stance; which by its nature includes accountability. While it is, admittedly, limited to those products that the two reviewers either get sent by publishers or purchase themselves (the latter probably stuff they are predisposed to like, since they're spending their own money), EN World has always had a pretty firm stance on reviews being not beholden to the publisher. I'd rather they stopped sending products than appease anyone with fake reviews.

Luckily, this does not happen any more. There were some incidents in the past of publishers attempting to coerce reviewers into good reviews, and I publicised them - posted the emails and everything. Zero tolerance. It stopped, and hasn't happened here in a long time.

As for the references to ads and reviews touched on in a couple of posts further back - in this industry, that's just nonsense. Maybe it happens with movies and stuff - I don't know. But in the decade I've been doing this I've never seen an ad/review related interaction (and, frankly, our ads are handles by an independent outside agency - I don't interact with the ad side at all).

Maybe some publishers stop advertising because of a bad review on a site that happened to be included on the network they advertise on, but I wouldn't know it if they did and wouldn't want to. Simply doesn't interact.

I should say, I can only talk about in my experiences. I have only wrote reviews for retailers, and I can strongly believe that the systems you guys have hear will ring in a better quality of review, because some of the places I worked had similar systems. (Paid reviews, accountability, No rating limitations etc). However, it feels the places I do write reviews for are pulling away from this type of system. Feels like its a matter of supply and demand. Everyone thinks they are a writer these days thanks to blogs, so there's a bigger supply of writers, and some just don't care about quality, so long as they can get the page views.

I know this was a big issue in the video game world a while back, where some publishers would take away advertising dollars from magazines that labeled a product unfavorably. However, in the RPG reviewing world, I have only had a few publishers write me about how much they hated my review. Those publishers simply elected not to send me review material anymore. They did not stop advertising nor did they stop selling their products.

 

[prosfilaes]

You naturally think 3-stars would be "this seller did everything OK; average review";

My problem with that is sells over the internet over a standardized platform like Ebay or Amazon is pretty trivial. Most sales should be perfect; I order the product at the listed price, it gets shipped promptly and arrives at a time dependent on the shipping I chose (and delays there are out of the control of the seller, usually) and then it arrives in the described condition. That's how it usually works. If I'm perfectly satisfied, why should I give less than 3 stars? In rating a RPG or video game or movie, there's almost something bad to note, but sellers normally completely satisfy me.

 

[Stormonu]

Oddly enough, when I'm looking at buying something, the first thing I look for is the "1 star" reviews. If the criticisms there strike a chord with me, I know to avoid that item. If it sounds like the reviewer is just a hater with an axe to grind, I feel generally confident that it must be a good product because folks can't come up with "real issues" about the product.

That isn't to say I won't read the 5 or 4 star reviews, but rather I tend to feel they are too gushing and willing to overlook real issues with the product. I also pretty much try to avoid something that can't maintain a 3.5 or better star rating.

 

[jaerdaph]

Good Old Fashioned Word of Mouth Advertising in the Internet Age

Wasn't always the case, but most RPG reviews are completely useless and meaningless to me now, especially the ones at RPGNow/DrivethruRPG (and in particular their "staff reviews").

Good buzz and discussion from everyday actual RPG consumers like myself about a game/product/setting/sourcebook/service/software etc. on the RPG message board circuit - that's what attracts me and sells me on a game now. And active, helpful publisher/developer participation in that same circuit goes a long way too - publishers that take the time to respond to their fan base and not make every post they make feel like "drive-by spam".

 

[Bedrockgames]

I don't think reviews have gotten any worse. If anything I think they are a bit better now. It is still mostly regular gamers writing the reviews so you always want to acount for style bias. I find places like EN World, TheRPGsite and onebook are all good (I like how onebook draws a distinction between featured reviewers and customer reviews).

 

[Bedrockgames]

Oddly enough, when I'm looking at buying something, the first thing I look for is the "1 star" reviews. If the criticisms there strike a chord with me, I know to avoid that item. If it sounds like the reviewer is just a hater with an axe to grind, I feel generally confident that it must be a good product because folks can't come up with "real issues" about the product.

That isn't to say I won't read the 5 or 4 star reviews, but rather I tend to feel they are too gushing and willing to overlook real issues with the product. I also pretty much try to avoid something that can't maintain a 3.5 or better star rating.

Honestly I am just as suspicious of a 1 star review as a 5 star review. Generally I ignore the rating system for rpgs. There just isn't enough volume for it to be reliable ( unlike say rotten tomatoes where they average out a ton of reviews). I find if you read the review and ignore the rating you get a much better sense of the product. I want to know why the reviewer liked or didn't like a product.