Reviews you can trust?

[Aberzanzorax]

Quite possible on the rose tinted nostalgia.

Maybe I should have said there were some better reviewers...as in there were plenty of bad reviewers, but there were some (Psion? I think he was pretty good IIRC) that were great.


Or perhaps that's rosey nostalgia as well...I am going from memory.

 

[RangerWickett]

I would love a Zero Punctuation-style RPG reviewer, who points out how every product is not nearly as good as it could be, even if it's on par with the rest of the gaming industry.

Then again, any such reviewer would probably suck compared to Yahtzee. The Escapist : Video Galleries : Zero Punctuation

 

[Umbran]

There were a lot of really bad reviews back then.

You may both be right, though.

Assume, for a moment, that Sturgeon's Law (90% of everything is crud) holds for reviews. If, today, there are 10 reviews, there'll be one good one.

If, in yesteryear, there were 100 reviews, there would have been 10 good ones.

If you think proportionately, there's no difference, but in absolute terms, there were more good reviews to be found in the past. If you have a good selection process for finding good reviews and avoiding bad ones, the past would have been better.

 

[enrious]

I'll agree with others that reviews seemed of higher quality 5-10 years ago. I wonder if that was because the market was "bigger" or less fragmented?

I'm thinking specifically of the d20 glut (and perhaps the "warning away" as much as the "enthusiastic sharing") and enworld reviews.

I sort of think part of it was there was a lot of "new and exciting" (cough) things coming out - to be sure there was a lot of junk, but there were things coming out by the big d20 publishers, which focused things (at least here).

[qoute]I found it very, very useful to read reviews here during the time period of, say 2002-2007.

I wonder if the decline in quality of reviews is due to the edition split, and the edition war (someone posting positive reviews because they like a system moreso than a product?).[/quote]

I also wonder if this was caused by the base-shift, but for a different reason. I can only speak for myself, but with the launch of 4e and subsequent heavy promotion of it here, I lost interest in this site.

No doubt Morrus would rightly say that ENWorld was just as committed to supporting 3.x players as the new 4e game and that if there was a reduction in 3.x content (say on the News page), it would have been due to a lack of actual 3.x news.

However, ENWorld simply lost a lot of relevance with me at that point and I have to wonder if I wasn't the only one.

Even when I moved from 3.5 to Pathfinder, I found few reasons to return here, instead finding more of what I wanted/needed over at the Paizo boards - although there the quality of reviews is dreadful. I suspect it's because of the way their review system is tied into the store, being product-based, but for whatever reason I've seen very few quality ones.

I also wonder if perhaps reviews are simply less needed now than before? During that time there were so many products, there was enough of a need for assistance in determining which were poor and which were great. Now it seems a small enough pool that I can do my own research without as great a need for advice.

Possibly. Or to take the 3.x/Pathfinder example, I already have more products than I could ever use, so the odds of me going out of my way to acquire a new one are slim. 10 years of products means a smaller list of products that will interest me. If I'm not unique, then less interest in buying = less interest in reviews.

Similarly, I wonder if number of products allowed for more of a "baseline" to be determined, whereas there isn't a clear point of middle of the ground now?

Interesting idea.

There's probably an element of rose-tinted-nostalgia going on there. I personally don't remember reviews being any better.

There were a lot of really bad reviews back then.

Oh, without a doubt. I tend to think of it more in terms of reviewers as opposed to reviews. I think back to the staff ENWorld reviewers and while I may not have agreed with every review, I valued their opinion.

With one or two exceptions, their reviews were knowledgeable (in terms of mechanics, but also from the standpoint of actually using the product - that means a lot to me in a review), and for whatever reason, the ones I'm thinking of seemed to be writing to me gamer to gamer.

Most of the reviewers I've read today seem to simply do nothing more than serve as an extended table of contents, with nothing more substantive than "Oh, I liked this" or "This is really neat" followed by a listing of how many pages per section. (Yes, I am exaggerating for effect)

So yeah, Morrus, there were bad reviews. I probably wrote some. But there were also good/great reviews by some recurring reviewers (here and elsewhere) and it seems like they've stopped reviewing (for any number of reasons).

The problem from my (limited) perspective is that there haven't been that many reviewers of the same quality to replace them.

Of course, the 64 thousand dollar question is - are there really that many people who miss good reviews?

Sadly, I'm not sure there are.

PS - Oh hey Morrus, fwiw, the D&D 4e and Pathfinder news/reviews I see up on the front page are the sort of reviews that I think are quality. Kudos to the two reviewers - I don't play 4e but I read the 4e news because it becomes interesting to me when it is so written.

 

[El Mahdi]

Honestly, I think rating things in that manner (numbers skewed towards the high side) has been common for, well...just about forever. I think it comes down to just common human nature.

If something "truly" bothers us, we're willing to give something a "bad" rating. But that tends to be rare. If something is just average, I think there's a predilection by most people to not want to "hurt" someone or something that hasn't hurt them, so they give it a 4 or 5 - even if subjectively, average should be a 3 or 2.

I've seen the same thing throughout my Air Force career. Every year (and sometimes more often) enlisted personel have an evaluation called an EPR (for Enlisted Performance Report). There are ratings on it for a number of different categories, and a section for comments, but the overall score is a rating of 1 to 5. The overall score is used in combination with promotion test scores (from two multiple-choice tests: one on the individual's specific career field, and one on general professional military knowledge), to determine who get's promoted. A 1 is for extremely poor performance and has a low point total applied to the overall testing score, and a 5 is for truly excellent performers with a high point total applied to the overall testing score. In theory, a 1 is for the absolute worst (completely or even criminally insubordinate, those convicted of a crime by the UCMJ, etc.), a 2 is for substandard performers, a 3 is for "average" performers, a 4 is for above standard performers, and a 5 is for "Exceptional" performers.

In practice, nobody wants to see their troops "hurt" when it comes to promotion chances, so the majority of ratings are 5's. Typically someone gets a 4 for being "average", and a 3 or lower for varying degrees of screwing up - with 3's fairly rare, and 1's and 2's extremely rare.

It's simple human nature.

The problem with today's "internet reviews" is that like everything else in our lives, we want things to be more and more "convenient". We want to be able to simply look at a numerical rating and "know" if we will like something or not...and that's just not practical.

If one wants to "know" if something might be to their taste, they have to "read" reviews. They have to read with a discerning mind, picking out the subjective from the objective. One has to do their homework by reading more than just one review on a product, and sometimes even reading up on the reviewer themself.

The best reviews are reviews that provide as much information as possible. I hate short reviews. And I hate reviews that are full of almost nothing but personal preferences and subjective opinions (I like x..., I hate y...) without stating why they do or don't like something (a reference so I can put their subjective statements in context).

Personally, I think most of the reviews here at ENWorld are absolutely awesome, but I tend to mostly ignore numerical ratings. I want to read what the reviewer thought and why. I haven't had an ENWorld review steer me wrong yet.

 

[KJSEvans]

TBH I mistrust 1 or 2 star ratings more than I mistrust glowing 5 star ratings because sometimes people have an axe to grind.

Actually I think the best way to look at a review is to read the content, not glance at the rating. If a person gives a detailed explanation as to why they love a book and it strikes a chord with me - i.e., sounds like he/she values the book for the same reasons I might - then I'll trust that. Even the axe grinders, as few as they are these days, are useful because they tend to find minor flaws and blow them out of proportion. If somebody gives a great book a 1 star review because it has typos, then I know it's a great book.

 

[gamerprinter]

Well, of course, I design Kaidan: a Japanese Ghost Story setting, adventures and supplements for PFRPG, as imprints under Rite Publishing. And while I can proudly say, the bulk of my product releases have earned 5/5 star ratings - I've gotten my share of 3 and 4 star ratings as well, even from reviewers who give other products of mine 5 stars.

Steve Russell of Rite Publishing regularly sends free copies of products to a host of reviewers, including the featured reveiwers of RPGNow, as well as Dark Mistress, Endzeitgeist, Dawn Fischer to name a few. As stated those reviewers seem honest enough to not always give me 5 star ratings. I have witnessed products (not mine) these same reviewers have given 1 or 2 stars, so I haven't seen reviewers with exclusive 5 star reviews.

Some of my more popular products seem to get reviews from not regular reviewers who want to give their take on what I create. One reviewer for my Frozen Wind free adventure (released Halloween last month) gave it high points for use in L5R games, even though it has designs for PFRPG only.

My experience is limited, as I haven't really followed reviews and reviewers until I started creating products of my own. But I'm firmly in the belief that 5 star reviews aren't exclusive of my products. Just enough 5 star reviews to know what I design seems to be on the right track - fitting the following that needs good feudal Japan/Asian horror content which Kaidan provides.

While I appreciate all reviews, Endzeitgeist, unlike the other reviewers reveal Spoilers every time, but he also tears every aspect of my products apart and comments on each portion. His reviews comb through every detail and he gives his point of view on each part. I always look forward to Endzeitgeist reviews due to the candor and specificity involved in each one.

Here's a link to an Endzeitgeist review of The Gift, my first adventure - to give you an idea of what I mean...

 

[enrious]

I agree, Enzeitgeist does reviews I like as well.

It's interesting, because some of the reviewers I've seen mentioned are exactly the ones I don't think are very good.

But hey, I think a review should be more than a table of contents with a number at the end.

 

[gamerprinter]

Another value to the reviews given by those like Endzeitgeist, is when they've mentioned problems or weaknesses in a given mechanic, I have gotten with my writer/designer, Jonathan McAnulty, after looking at the specific product and reviewed the rules on d20pfsrd.com to verify and made changes that in a couple days get's rereleased as an update to the product. End has even updated reviews based on these changes.

So a good review can get some publishers to fix problem areas right away, so it becomes a better product.

 

[Jhaelen]

Actually I think the best way to look at a review is to read the content, not glance at the rating.

This! The actual rating is often meaningless because there's so much variance in the reviewer's and the reader's expectations and tastes

Even ratings given to products with objectively measurable qualities are often meaningless to a reader because the test results may be weighted in ways that don't represent a reader's preferences. In such cases it's important that the review clearly shows how the ratings were calculated.