Reviews you can trust?

[PaulofCthulhu]

The way I personally approach it, reviews simply tell me the opinion of a singular product by a singular person. Different people can have different likes/dislikes.

So reviews work when the reviewer you're reading has a similar outlook to your own on a number of things. That only comes from validating your experience of the same product.

Unfortunately, as far as I can tell, there aren't many regular reviewers these days (there are some) and it's not always easy to peg their mores vs. mine.

Also I suspect many reviews are "read-thoughs" rather than "play-throughs" - and there can be a big difference between the two (esp. if you're looking for something to play)!

 

[IronWolf]

I do some reviews on my blog. Looking back through the handful of them it seems I tend towards the high side, though I do have a 2.5/5 on there as well from a company I normally find quality product from. In my case I review items I have purchased. I tend to buy products I think will like so I think that leans my ratings to the higher side of the spectrum.

I do attempt to provide detail on the product within the post and highlight what I did and did not like. That way readers can hopefully form their own opinion and realize what I might have liked might not be something that matches their tastes and would alter their own tankard rating to their tastes.

I think there are still good reviewers out there, it is just a matter of finding them. For reviewers that have some type of commenting system, let them know what you think. I know I would be happy for constructive comments on how to improve my reviews if they would make them more helpful to the community.

 

[Dark Mistress]

As someone that does review a lot and I am sure I am one of the people being discussed. I personally am surprised as many people seem to like my reviews as they do, yet I keep getting people not just publishers asking me to review stuff*shrug*. I personally prefer to give a bullet point review of here's what you get and let people make up their own minds. The rating system is flat flawed IMHO. The reason i say that is it really comes down to a matter of taste on what the reviewers likes and dislikes are. I try to minimize that with my reviews but it is still true. I think reviews would be better and more helpful with out a rating system personally.

I do the bullet point review because that's what I want to know. I care more about what all is in a product, less about what the reviewer thought of it. Plus I fully admit I don't care a ton about the rules. To me if something plays well/works and is interesting, then I am happy even if it breaks a rule to do it. End on the other hand tends to care more about the rules. I also don't like to give spoilers while End doesn't mind.

As for the lack of low reviews, that's true and I dislike doing low reviews. Not for the reasons some might think though, plus what each of us consider a rating varies. For one I normally have trouble finishing reading something I would give a low review to and two the main reason i do reviews is to inform people and help the hobby, by pointing people towards the stuff I think is good. Which helps sales (hopefully) which increases the odds of more products being made.

For those curious how I view the rating system.
5 star = Product with lots of useful and interesting idea's for a reasonable price.
4 star = Product like a 5 star but needs a little more polish or it has some good and interesting stuff and some filler and/or stuff that needs work.
3 star = solid product with some cool bits but either needs a lot of polish/work or works as intended but nothing special. and/or a 4 star priced way to high.
2 star = Needs to much work to be usable, which often means I never finish reading it. and/or a 3 star that is priced way to high which is usually the only ones this low I finish reading.
1 star = waste of my time to read it, which means I never finished reading it.

As for using reviews what I would do and I think is best since reviewing is very subjective. Is look at the stuff you own, go check the reviews. If you can find a reviewer that seems to give a similar rating to a book as you would give on a lot of stuff then you found someone that has similar style and taste and just pay attention to them.

But then if you are lamenting about a lack of good reviews, maybe you should start doing your own reviews and doing them the way you think they should be done.

 

[DonTadow]

As a reviewer, I 100% agree with the original poster.

I started reviewing 7 years ago. When I started, there were a pool of us reviewers who prided ourselves on keeping our average review number closer to 6 (on a scale of 1 to 10). The closer you were the 6 the more fair you were and the more important your opinion was when you did give the 8 or 9.

When I first started reviewing, I went through an interview process, had to submit writing samples and even had a 2nd round of grilling.

That is not being done these days. There's one reviewer who I'm ashamed to call a colleague, because I've never seen her give anything lower than a 9. Add a bunch of novice writers who don't understand how to be critical to sites who would prefer you write closer to marketing text than critiquing the product. It's akin now to having everyone on the Little League Baseball Team be called a winner.

Because review sites are moving towards marketing text as opposed to critiquing the product, reviewers are urged to not write a review if you think it is going to get a 5 or below. Thus, anything that should be given a 1, 2 or 3 gets discarded by the reviewer. I read a dozen or so products a month, I only write reviews for 3 or 4. There are times where I give products a 5 when I feel it should receive a 3 or 4, simply because my hands are tied.

I disagree that a scale system for reviewing is flawed. I've learned from the best. A good reviewer can distinguish his personal opinion from strong characteristics of a product. There have been a number of products that I don't like personally, but were good products based on the objective they were trying to achieve. I think its fairly lazy writing to just regurgitate facts about the product that are already listed in the marketing text by the publisher. At this point you've told the reader nothing.

 

[Morrus]

I'd argue otherwise, [MENTION=22622]DonTadow[/MENTION].

EN World, for example, has moved towards a paid reviewer stance; which by its nature includes accountability. While it is, admittedly, limited to those products that the two reviewers either get sent by publishers or purchase themselves (the latter probably stuff they are predisposed to like, since they're spending their own money), EN World has always had a pretty firm stance on reviews being not beholden to the publisher. I'd rather they stopped sending products than appease anyone with fake reviews.

Luckily, this does not happen any more. There were some incidents in the past of publishers attempting to coerce reviewers into good reviews, and I publicised them - posted the emails and everything. Zero tolerance. It stopped, and hasn't happened here in a long time.

As for the references to ads and reviews touched on in a couple of posts further back - in this industry, that's just nonsense. Maybe it happens with movies and stuff - I don't know. But in the decade I've been doing this I've never seen an ad/review related interaction (and, frankly, our ads are handles by an independent outside agency - I don't interact with the ad side at all).

Maybe some publishers stop advertising because of a bad review on a site that happened to be included on the network they advertise on, but I wouldn't know it if they did and wouldn't want to. Simply doesn't interact.

 

[Treebore]

The only reason I wrote reviews of products is I liked them. My time was, and is, too valuable to waste writing about "average", or even "crappy" products. Were they "fair", "balanced", no idea. I never received training.

 

[Dice4Hire]

I treat reviews higher than I would a dozen or so comments on a messageboard, but I do not consider them the be-all and end-all of what I buy.

Still, it is good to see reviews, and I try to read them for most products I am undecided on. Enworld's reviews are pretty good, overall.

 

[OnlineDM]

I've written a few reviews on my blog, but only for products that I purchased myself and read all the way through. I haven't assigned any "point ratings" or anything like that; I've just written my thoughts on the product (what I liked, what I didn't like).

I gave Reavers of Harkenwold a glowing review, because I really liked it. My review of Heroes of the Feywild was largely positive, too, for similar reasons. My review of Heroes of Shadow was unfavorable, and I explained it was because I probably wasn't the target audience for the book. My review of Heroes of the Fallen Lands was favorable, again, because I enjoyed the book.

So, three out of four reviews were positive and one was mildly negative. Not a big surprise, considering that I only purchased books that I expected to like.

I also purchased The Shadowfell: Gloomwrought and Beyond during this time period and have not reviewed it. This is because I didn't enjoy the book all that much, still haven't finished reading it, and didn't feel like exerting more effort to say "Meh". I imagine this is a common occurrence with reviewers; it's hard to get up the energy to write a review that's mediocre.

Edit: I just remembered that I also reviewed a third-party product whose producer had put out a call for reviews. I received an advanced copy of the PDF, informally acted as a copy-editor, and then posted a review on the blog. I'd call it an overall positive review, though certainly not gushing (the product wasn't really the sort of thing I'd personally use in my own games).

 

[SiderisAnon]

When I look at reviews for products, I find three problems that make it impossible to trust many reviews.

1) Somewhere along the line, a three-star "average" turned into a three-star "you suck" with five star minimum. I'm not sure when this happened, but it's there. Companies on sites like E-Bay and Amazon get really upset when someone gives them anything less than five star; I've seen reports of people being contacted by companies to find out what they did so terribly wrong that they only got three or four stars.

2) Reviews for many products are tainted with fake reviews, both for and against. I don't see this as much with gaming material, but anyone who has ever shopped for technology and pays attention to the reviews will certainly have seen this.

3) Writing a good review should mean you took the time to at least finish reading the product. If it's bad, you probably won't finish it. Even worse for the chances of getting an honestly negative review is that so many products are weeded out because the product is so awful looking in the first place. A great example of this is RPGNow, where there are numerous really bad looking products with no reviews; the people who would write bad reviews already chose not to but it at all from the description and demo. Some of those full-sized previews are enough example to know the product would get a one star review at best.

There are also a lot of useless reviews. Many contain no real information about the quality or contents of the product. (I recently was looking at a product on RPGNow where one of the reviews was just a bunch of random characters, presumably just enough to get the system to accept the score given.)

I do occasionally find some really good and detailed reviews, and these will sway my opinion more than anything else. During the recent sales, I bought a set of spells for my Pathfinder game because one reviewer had done a solid, detailed analysis of each of them and I now had enough information to make an informed decision. (Too many of the RPGNow product pages give little or no information on the contents of the book.)


I keep wanting to write reviews for various products I use. I am building a new campaign world and have been making notes on the various books I'm using for it because it's a segment of the PDFs that I think there aren't enough reviews in. The problem comes down to that I can spend and evening writing an intelligent review, or I can spend the evening adding to my campaign, and the campaign wins.

 

[prosfilaes]

The only reason I wrote reviews of products is I liked them. My time was, and is, too valuable to waste writing about "average", or even "crappy" products. Were they "fair", "balanced", no idea. I never received training.

Heh. When I've wasted my time and money on a book, it helps to vent my spleen on a good open savaging of the book