Reviews v. Ennies

[Gizzard]

JoeGKushner's review thread got me to thinking about how I interpret online reviews (and reviewers); it also got me to thinking about this one specific issue:

Though a Freeport fan, I put off buying Black Sails because of mediocre reviews here (3 at 3.33). Yet, I see that Black Sails not only was nominated for best module, but won the Ennie for that category. (This is the only category I was curious about, so YMMV for other categories...)

Anyway, I am wondering what this discrepancy means in terms of the value of reviews/awards for D20 products? Obviously the Ennies voters didn't read the reviews before choosing their favorites... ;-)

 

[EricNoah]

Could be sample size -- even a product with lots of reviews doesn't really give you a decent sample if we're just going by "counting stars."

And it could be the star rating system, too -- one person's 3 might be another person's 2, etc. Three-point-something isn't all that bad a score on this scale, necessarily.

 

[Psion]

Actually Black Sails earned a silver. The Gold was Lost city of Barakus.

You have two things going on here:

- The final phase is a people's choice phase. Market penetration and sheer popularity play a big part.
- Simple subjectivity. 3 reviewers is a small sample size, and we may rate them different than some people do.

You can also think of it like a movie that isn't a hit with the reviewers but is a box office smash.

(Speaking for myself, I very nearly gave BSoF a 4. In fact, I did originally. It was a fairly strong 3.)

 

[trancejeremy]

It's also likely many people rated BSOF without having actually read it or owned it. Companies with good reputations often get the benefit of the doubt.

It's also probably a bit better than the reviews it got here. Sometimes things suffer a bit because you have to choose between a 3 and a 4. It's probably a 3.5, because IMHO, it's not great (a 5) , but it's pretty good (a 4), but has enough minor problems with it to drop it down a bit.

(I think I gave it a 7 when I voted for it in the Ennies)

 

[Chaos Drake]

The ENnies could be more prominent. Where do I look for previous ENnie winners?

 

[rounser]

Market penetration and sheer popularity play a big part.

The reviews for The Return to the Temple of Elemental Evil make a solid case for your point, I think.

 

[Ron]

It's also likely many people rated BSOF without having actually read it or owned it. Companies with good reputations often get the benefit of the doubt.
[...]

Trancer Jeremy is right. Because of this, review's rating is usually a more reliable indicator of the voter opinion, as it is less likely that he or she assigned it without reading the product. The problem with reviews is that, for most products, there aren't enough of them to the mean became close to true estimative of the real average opinion about the product.

What I usually do is read a few reviews of different grades. A good review would allow you to know if you will like the product independent of the reviewer's grade.

 

[Ron]

The reviews for The Return to the Temple of Elemental Evil make a solid case for your point, I think.

If I ever get over my revulsion to that module and the year I lost playing it, I may write a review to help placing the average review grade where it really should be.