MetaPost comments and questions about the messageboards and other parts of EN World. If you have a problem, this is where to go. Moderator email addresses are posted here.
* Read all statements of X should be Y as "IMHO X should be Y" *
Yesterday I couldn't get into the boards so I went through and started reading reviews for d20 products. This got me thinking about another thread (http://enworld.cyberstreet.com/showt...threadid=27439) in which Avalanche Press complains about getting stinky reviews. All other aspects of that issue ASIDE, I became interested in whether or not Avalanches reviews on ENworld are in fact favorable.
Morrus says: Of course, this is made funnier by the fact that said publisher has received two 4/5 scores from EN World staff reviewers, a 3/5, along with two 2/5 scores. Average score = 3/5, which does not constitute,as they put it, "disturbingly unfavorable reviews
our products received up to, and including XXXXXX".
So I went through the three official reviewers, Kushner, Simon and Psion. (heh, Simon and Psion, anyone remember Simon and Simon?)
Scrolling down quickly I found that out of Psions 100+ reviews not a -single- one warranted a "1 - atrocious" rating. Kushner as well has not nailed anyone with a 1 rating. In Simon's 100+ reviews there are only about 3 or 4 atrocious ratings.
(i can't get back to the reviews page right now, but when I do i'll give a more statistically accurate rundown). In addition 2's are much more rare and 5s and 4s. So -in effect-, even though Avalanche's rating is 3.00, this is actually not average as is stated, but is actually 1 step away from being put in the same catagory as those companies who released one bad .pdf and got slammed.
Now, in my opinion, you should ideally be able to add up all the scores from all the products, divide by number of reviews and get 3.00 . So when I get access to the reviews page that is what I am going to do. My guess is that the average is much closer to 3.66 than 3.00
"What does this matter?" you ask? Well it doesn't, but I still find the situation interesting. This is in no way to impinge on the text of the reviews (which I find excellent by all reviewers, they have firmly convinced my of quintessential wizard and some green ronin stuff!)
Here is a question for the rest of you: Why does this always seem to happen, in grades, movie reviews, etcetera. A "C" should be the average grade in classes, but often B is the average. If someone gets an A in a class they should be in at least the top 10 if not 5 percent.
Why, time and again will people say a movie is "nothing special" and then rank it 7 on a scale of 1=10? A 7 should be a pretty darn good movie. (my theory, people are mentally affected by the way grades work; 7=70 to 79% which was a C in my high school)
So here is what I think: if there are only 3-4 atrocious ranks, there should only be 3-4 amazing ranks. that would surely make them stick out of the crowd much more. As it is, the rankings for publishers are absolutely clogged between 3 and 4.5 while the low numbers are virtually untouched. Spread it out a bit.
Opinions, thoughts, rants, flames?
__________________ have you ever retired a human by mistake?
My opinion would be to get rid of the ratings altogether and let the reviews stand on their own. Because :
1) This would stop people (if any have done this) from posting a short review just to pad the ratings.
2) A number rating is too subjective. For instance, some people deduct points or fractions thereof for artwork, which other people might not care about. Some people just want good crunchy or crunchy and story. Unless you had separate ratings for various parts of a review (artwork, grammar, crunchy, story, etc).
2) IMO a 1-5 rating does not give enough leeway if ratings were wanted.
I've done 19 products so far. THe numbers break down as:
5/5 : 2
4/5 : 11
3/5 : 5
2/5 : 1
1/5 : 0
I have reviewed two products I got for free, everything else I bought. I try to only buy good products, so that's why so many of the books I review get a 4. Also, it's more enjoyible for me to review a book I like verse a book I don't like. I think those tweo reason are why you see more higher reviews then lower.
Originally posted by tleilaxu Here is a question for the rest of you: Why does this always seem to happen, in grades, movie reviews, etcetera. A "C" should be the average grade in classes, but often B is the average. If someone gets an A in a class they should be in at least the top 10 if not 5 percent.
There's more than one way to grade. One way to say "The average will be a C". The other is to say, "The basic acceptable grade is a C". These compose two different grading philosophies.
What you want - the average being C - is a mathematical construction that you can't easily get unless you have all the scores together before you actually assign the final grade. You grade on some other numeric scale, find what the avverage is, and call that a C. If you're doing one grade at a time, and adding them to a pile, it's not realistic to look for this. Especially if general quality rises over time. I guess you could have the computer constantly recomputing what's "average", but that's a pain, because you have to say, "averaged with respect to what?" Averaged with respect to all reviews of all products? With respect to all reviews of this one product? With respect to all reviews by this reviewer?
On top of that, it isn't actually useful in this context. In order to find out what the "average is C" really means, you have to go through and read all the reviews. What happens if you apply the "C=average" to a whole lot of products that are actually really, really good? You get a misleading result.
The "C is the basic acceptable" grade is more user-friendly, easier for the reader to interpret. Reviews are not math test grades. Leave 'em be.
Last edited by Umbran; 19th October 2002 at 05:09 AM..
I'm not suggesting that we somehow fit all new reviews into a mathematical model, I am just suggesting that in no way does "3" equate to a average rating relative to other products and that reviewers shouldn't hesitate to really nail a product if it isn't as good as others.
__________________ have you ever retired a human by mistake?
Originally posted by tleilaxu I'm not suggesting that we somehow fit all new reviews into a mathematical model, I am just suggesting that in no way does "3" equate to a average rating relative to other products and that reviewers shouldn't hesitate to really nail a product if it isn't as good as others.
I think reviewers do nail products they think are bad. I also don't think you can really speak in general terms like this and get any concrete discussion going. All the reviewers use different criteria when reviewing an item. There really isn't even a uniform amount of info in the review. I've see some reviews that barely give any real info and it's mostly their opinion. Personally, I think it would be far more productive to get a few people to go back through all the reviews and start deleting the ones that are not up to code. The standard of reviews has increased, I think it should be applied to all of them.
Here is a suggestion that I just thought of that I think could make the ratings more accurate. Just include an extra column to the right of the star rating with the percentile rank of a product. (For example: Tomes and Tunnels has a 3.7 star average, which ranks it in the 61st percentile of all products reviewed on ENworld. (meaning it has a higher score than 61% of the products) us.imdb.org has some system for including number of votes/reviews into this to add weight (IE if 10 people say 4.5 it means more than if 2 people say 5.0)
What do you all think of this suggestion? I know that the admins have other stuff to worry about right now but this kind of addition wouldn't take to long to html in. Heck, I could do it on a spreadsheet and email you the results if you wanted. It could even be made to auto-update the numbers when new data is added.
__________________ have you ever retired a human by mistake?
Originally posted by Crothian Personally, I think it would be far more productive to get a few people to go back through all the reviews and start deleting the ones that are not up to code.
which code, Crothian?
if you're going to institute some sort of review standards (i.e. a code), i'm with dragongirl, toss the 1-5 rating system
__________________ So THAT'S what an invisible barrier looks like!
I think reviewers do nail products they think are bad. I also don't think you can really speak in general terms like this and get any concrete discussion going. All the reviewers use different criteria when reviewing an item. There really isn't even a uniform amount of info in the review. I've see some reviews that barely give any real info and it's mostly their opinion. Personally, I think it would be far more productive to get a few people to go back through all the reviews and start deleting the ones that are not up to code. The standard of reviews has increased, I think it should be applied to all of them.
Well, my intention is not to generalize, I purposely went through and read the reviews for products that got 1, because you can learn as much about what -not- to do as the opposite. My intention is not to discuss this, but merely about the proponderance <sp?> of 4s and 5s and the general lack of 1s and 2s. If you scroll down a page and see twenty 5s and 4s, ten 3s, and one 2 guess what: The 2 -is- atrocious, at least from someone looking at the page and trying to decide what to buy. I wouldn't touch any products that got a 2 on ENworld because they are so rare they -must- be bad to warrant that rating.
I think your suggestion for culling empty reviews is one that deserves some attention. I read more than one review that basically said 'woopie this product is awesome end of story'.
edit:
BTW Crothian, your average rating is 3.74
__________________ have you ever retired a human by mistake?
Last edited by tleilaxu; 19th October 2002 at 05:34 AM..
Originally posted by tleilaxu
BTW Crothian, your average rating is 3.74
It'll drop. The next review I'm doing is a 3/5. After that I'm going to do some of my favorite companies books that I haven't reviewed yet. They are my least favorite of their books, so will probably be a pair of 3's and a single 2 in there.
A product would have to be pretty darn bad for me to give it a 1. Just like a product has to be darn amazing to get a 5. If I ever get around to doing the 3 core books, they will all be 4's. There fine books, but not 5's in my so very humble opinion.
There is no "code" for reviews to my knowledge. And I looked for one to start writing a review myself. You want to propose one, Crothian?
I don't know if dropping the 1-5 rating system is really a great idea. Sure, it's hardly precise, but people aren't looking for precision from such a thing. They want a quick guideline.
Originally posted by Umbran There is no "code" for reviews to my knowledge. And I looked for one to start writing a review myself. You want to propose one, Crothian?
I don't know if dropping the 1-5 rating system is really a great idea. Sure, it's hardly precise, but people aren't looking for precision from such a thing. They want a quick guideline.
If a code was instituted it would be up to the 3 staff reviewers to set. I'm not going to step on their toes as they have a much better understanding for the review process then little old me. However, if they wanted suggestions I'm sure I could come uyp with a few.
And the rating system isn't going anywhere. It's coded into the program and I don't think anyone wants to try to alter or try to remove it. I'm not sure they can.
Edit: the closest we have to a code is the review FAQ. Please read it before posting as it has information you do need to know.
For me, there are some things I try to keep in mind.
1. Does it do what it sets out to?
2. How's the layout & art?
3. How's the editing?
4. How's the text to page ratio? I despise companies that put one page for table of contents, another page for credits, another two pages for OGL. One page for toc and credits, another page for OGL. That' it!
5. If the product doesn't suit my taste, will it others?
6. How expensive is the product?
7. Are there other products in the same field?
I don't think I've had a 1 product yet and some products, maybe should've been higher. Wrath & Rage for DMs is a 4 but players, a 2.
I also don't love the 1-5 system, but I'm stuck with it here and on Gaming Frontiers. I think 1-10 would probably be better.
Originally posted by Blacksway Purely for statistical freaks out there...
Hmm... As you said, these reviews don't differentiate between "bought and liked" and "got sent a copy to review." Is it possible - from the database - to discern which are which? Otherwise, these statistics don't help much...
(Not that I think that this entire topic is necessary at all, but since you guys are discussing it...)
Originally posted by tleilaxu I just want to reiterate that I think the reviewers are doing a fine job for ENworld, but there is nothing wrong with trying to improve the system.
To infinity and beyond
Absolutely.
For example, I'd prefer a 1-10 scale to the current 1-5 one.
Originally posted by tleilaxu Here is a suggestion that I just thought of that I think could make the ratings more accurate.
Ooh, my buttons are getting pushed
A thing like this does not increase accuracy. It increases precision. There's a difference.
Accuracy talks about how correct the number is, how well it reflects reality. Precision, in essence, talks about how many decimal places the measurement can realistically include.
A laser range finder can take measurements very precisely. It can always give you a number out to many decimal places. Your measurement may still be inaccurate if you use this very precise tool incorrectly (by saying you measured the width of a field when you actually measured it's length, for example).
There is no point to taking a measurement precisely if you aren't going to be very accurate anyway. That's the situation we have here. The rating system is subjective (since it's rating an opinion, and opinions are subjective). There's no "code" for how you give a rating. Unless everyones giving ratings in exactly the same way, for the same reasons, you're not getting a very accurate measurement. There's little point in going for high precision on it.
In other words, the ratings system is by it's nature very vague. Giving more info on the ratings is then not terribly useful. It actually then makes sense to keep the ratings vague, rather than giving percentile scores, or going to a 1 to 10 system. You actually have a greater chance of getting consistent accurate measurements if you give people less freedom of "false precision".
Also, how many ratings do most products get? Percentile scores are not particularly good if the number of scores is less than (you guessed it) 100 or so. If only 5 people rate a product, a percentile score is pretty meaningless. If there are many such products, and you weight for how many people actually rated the product, you now reduce many of the rating to statistical irrelevance. If you're throwing many of them away, what's the point of doing a percentile score in the first place?
In the end, it comes down to this - what do users want to get out of reviews? They want to know how good the product is. Percentile socres don't actually tell you that. They are a bit of statistical minutae that only tells you something about the products if their quality actually follows a normal distribution.
(edit: My typing and grammar are often rather inaccurate )
Last edited by Umbran; 19th October 2002 at 05:27 PM..
Originally posted by Umbran
There's no "code" for how you give a rating. Unless everyones giving ratings in exactly the same way, for the same reasons, you're not getting a very accurate measurement. There's little point in going for high precision on it.
When I say code, I'm talking more about some minimium requirements for the review. Sort of like a word length of a couple hundred words. There are many reviews of about 4 lines or so, and I haven't read any of that length that get the job done.
Reviewers don't review "everything" -- they get to pick and choose what they review. Maybe they stay away from the stinkers, whether consciously not. I certainly would rather spend my time getting to know a good or average product than a bad one.
When I look through the product lists, I find that not only do I avoid products with low scores -- I also avoid products that have been out for a while and have not garnered even one review. To me that says a) people aren't buying it or b) people didn't care for it.
Linear Mode
And yet another word from our sponsors
