Review inflation on ENworld

[tleilaxu]

* Read all statements of X should be Y as "IMHO X should be Y" *

Yesterday I couldn't get into the boards so I went through and started reading reviews for d20 products. This got me thinking about another thread (http://enworld.cyberstreet.com/showthread.php?s=&threadid=27439) in which Avalanche Press complains about getting stinky reviews. All other aspects of that issue ASIDE, I became interested in whether or not Avalanches reviews on ENworld are in fact favorable.

Morrus says: Of course, this is made funnier by the fact that said publisher has received two 4/5 scores from EN World staff reviewers, a 3/5, along with two 2/5 scores. Average score = 3/5, which does not constitute,as they put it, "disturbingly unfavorable reviews
our products received up to, and including XXXXXX".

So I went through the three official reviewers, Kushner, Simon and Psion. (heh, Simon and Psion, anyone remember Simon and Simon?)

Scrolling down quickly I found that out of Psions 100+ reviews not a -single- one warranted a "1 - atrocious" rating. Kushner as well has not nailed anyone with a 1 rating. In Simon's 100+ reviews there are only about 3 or 4 atrocious ratings.

(i can't get back to the reviews page right now, but when I do i'll give a more statistically accurate rundown). In addition 2's are much more rare and 5s and 4s. So -in effect-, even though Avalanche's rating is 3.00, this is actually not average as is stated, but is actually 1 step away from being put in the same catagory as those companies who released one bad .pdf and got slammed.

Now, in my opinion, you should ideally be able to add up all the scores from all the products, divide by number of reviews and get 3.00 . So when I get access to the reviews page that is what I am going to do. My guess is that the average is much closer to 3.66 than 3.00

"What does this matter?" you ask? Well it doesn't, but I still find the situation interesting. This is in no way to impinge on the text of the reviews (which I find excellent by all reviewers, they have firmly convinced my of quintessential wizard and some green ronin stuff!)

Here is a question for the rest of you: Why does this always seem to happen, in grades, movie reviews, etcetera. A "C" should be the average grade in classes, but often B is the average. If someone gets an A in a class they should be in at least the top 10 if not 5 percent.
Why, time and again will people say a movie is "nothing special" and then rank it 7 on a scale of 1=10? A 7 should be a pretty darn good movie. (my theory, people are mentally affected by the way grades work; 7=70 to 79% which was a C in my high school)

So here is what I think: if there are only 3-4 atrocious ranks, there should only be 3-4 amazing ranks. that would surely make them stick out of the crowd much more. As it is, the rankings for publishers are absolutely clogged between 3 and 4.5 while the low numbers are virtually untouched. Spread it out a bit.

Opinions, thoughts, rants, flames?

 

[Dragongirl]

My opinion would be to get rid of the ratings altogether and let the reviews stand on their own. Because :

1) This would stop people (if any have done this) from posting a short review just to pad the ratings.

2) A number rating is too subjective. For instance, some people deduct points or fractions thereof for artwork, which other people might not care about. Some people just want good crunchy or crunchy and story. Unless you had separate ratings for various parts of a review (artwork, grammar, crunchy, story, etc).

2) IMO a 1-5 rating does not give enough leeway if ratings were wanted.

All I can think of at the moment.

 

[Crothian]

I've done 19 products so far. THe numbers break down as:

5/5 : 2
4/5 : 11
3/5 : 5
2/5 : 1
1/5 : 0

I have reviewed two products I got for free, everything else I bought. I try to only buy good products, so that's why so many of the books I review get a 4. Also, it's more enjoyible for me to review a book I like verse a book I don't like. I think those tweo reason are why you see more higher reviews then lower.

 

[Umbran]

Here is a question for the rest of you: Why does this always seem to happen, in grades, movie reviews, etcetera. A "C" should be the average grade in classes, but often B is the average. If someone gets an A in a class they should be in at least the top 10 if not 5 percent.

There's more than one way to grade. One way to say "The average will be a C". The other is to say, "The basic acceptable grade is a C". These compose two different grading philosophies.

What you want - the average being C - is a mathematical construction that you can't easily get unless you have all the scores together before you actually assign the final grade. You grade on some other numeric scale, find what the avverage is, and call that a C. If you're doing one grade at a time, and adding them to a pile, it's not realistic to look for this. Especially if general quality rises over time. I guess you could have the computer constantly recomputing what's "average", but that's a pain, because you have to say, "averaged with respect to what?" Averaged with respect to all reviews of all products? With respect to all reviews of this one product? With respect to all reviews by this reviewer?

On top of that, it isn't actually useful in this context. In order to find out what the "average is C" really means, you have to go through and read all the reviews. What happens if you apply the "C=average" to a whole lot of products that are actually really, really good? You get a misleading result.

The "C is the basic acceptable" grade is more user-friendly, easier for the reader to interpret. Reviews are not math test grades. Leave 'em be.

 

[tleilaxu]

I'm not suggesting that we somehow fit all new reviews into a mathematical model, I am just suggesting that in no way does "3" equate to a average rating relative to other products and that reviewers shouldn't hesitate to really nail a product if it isn't as good as others.

 

[Crothian]

I'm not suggesting that we somehow fit all new reviews into a mathematical model, I am just suggesting that in no way does "3" equate to a average rating relative to other products and that reviewers shouldn't hesitate to really nail a product if it isn't as good as others.

I think reviewers do nail products they think are bad. I also don't think you can really speak in general terms like this and get any concrete discussion going. All the reviewers use different criteria when reviewing an item. There really isn't even a uniform amount of info in the review. I've see some reviews that barely give any real info and it's mostly their opinion. Personally, I think it would be far more productive to get a few people to go back through all the reviews and start deleting the ones that are not up to code. The standard of reviews has increased, I think it should be applied to all of them.

 

[tleilaxu]

Here is a suggestion that I just thought of that I think could make the ratings more accurate. Just include an extra column to the right of the star rating with the percentile rank of a product. (For example: Tomes and Tunnels has a 3.7 star average, which ranks it in the 61st percentile of all products reviewed on ENworld. (meaning it has a higher score than 61% of the products) us.imdb.org has some system for including number of votes/reviews into this to add weight (IE if 10 people say 4.5 it means more than if 2 people say 5.0)

What do you all think of this suggestion? I know that the admins have other stuff to worry about right now but this kind of addition wouldn't take to long to html in. Heck, I could do it on a spreadsheet and email you the results if you wanted. It could even be made to auto-update the numbers when new data is added.

 

[Mr Fidgit]

Personally, I think it would be far more productive to get a few people to go back through all the reviews and start deleting the ones that are not up to code.

which code, Crothian?

if you're going to institute some sort of review standards (i.e. a code), i'm with dragongirl, toss the 1-5 rating system

 

[tleilaxu]

I think reviewers do nail products they think are bad. I also don't think you can really speak in general terms like this and get any concrete discussion going. All the reviewers use different criteria when reviewing an item. There really isn't even a uniform amount of info in the review. I've see some reviews that barely give any real info and it's mostly their opinion. Personally, I think it would be far more productive to get a few people to go back through all the reviews and start deleting the ones that are not up to code. The standard of reviews has increased, I think it should be applied to all of them.

Well, my intention is not to generalize, I purposely went through and read the reviews for products that got 1, because you can learn as much about what -not- to do as the opposite. My intention is not to discuss this, but merely about the proponderance <sp?> of 4s and 5s and the general lack of 1s and 2s. If you scroll down a page and see twenty 5s and 4s, ten 3s, and one 2 guess what: The 2 -is- atrocious, at least from someone looking at the page and trying to decide what to buy. I wouldn't touch any products that got a 2 on ENworld because they are so rare they -must- be bad to warrant that rating.

I think your suggestion for culling empty reviews is one that deserves some attention. I read more than one review that basically said 'woopie this product is awesome end of story'.

edit:
BTW Crothian, your average rating is 3.74

 

[Crothian]

BTW Crothian, your average rating is 3.74

It'll drop. The next review I'm doing is a 3/5. After that I'm going to do some of my favorite companies books that I haven't reviewed yet. They are my least favorite of their books, so will probably be a pair of 3's and a single 2 in there.

A product would have to be pretty darn bad for me to give it a 1. Just like a product has to be darn amazing to get a 5. If I ever get around to doing the 3 core books, they will all be 4's. There fine books, but not 5's in my so very humble opinion.