Re. EN World staff reviewers and good/bad reviews

[CRGreathouse]

I think the idea of removing the top and bottom scores is a good one; either that or just using the most common score (is that mode or median?)

The most common score is the mode; the median is the score in the middle. The system discussed on the Meta forum was (mean * median) ^ .5, the square root of the product of the mean and median. This would give good results without entirely discounting the extreme scores.

 

[hong]

If I'd thought of that 18 months ago, I'd have done it. Problem now is that there are over 1000 reviews already done and scored, and no way to "port" the scores over to such a system.

I think the idea of removing the top and bottom scores is a good one; either that or just using the most common score (is that mode or median?)

[statsgeek]

The mode is the score that appears most frequently. The median is the score such that 50% of reviews are below it, and 50% are above.

I would tend to use the median, because it provides more information. Suppose you have 200 reviews that score 3/5, and 201 reviews that score 4/5. In this case, the mode is 4. If however the next review you get is 3, the mode drops to 3.5 all of a sudden (assuming you take the average of the two). Depending on what the next review is, it might move back up to 4, or down to 3. Sudden shifts in an index are usually not considered desirable.

[/statsgeek]

 

[trancejeremy]

Well, don't forget there are two points to having reviews - one, as press for the company who made the product, but more importantly, to let people know how good a product actually.

For those of us who actually pay for RPG stuff, and don't get them for free, reviews are a god-send.

A lot of the staff reviews on some sites (not here!, and not at that DM's Havan's site, but a few others) seem to be little more than PR fluff...

And I do think, that getting a product for free does skew the review a bit. If you get a bad (or even average) product for free, you're going to be less upset with it than if you shelled out your hard earned money for it.

 

[Lady Dragon]

I think a big problem with being liberal about handing out 4's and 5's is that when something really remarkable comes out there's know way to distinguish it from all the other products that have been given 5's.Perhaps a scale of 1-10 would be better thus making 5 and 6 average 7 and 8 above average 9 excellent and save the 10's only for the best of the best.

And taking out the high and low would help get rid of bias.

 

[Psion]

Re: Re: r/t ENworld reviews

One possible solution would be to review products like a term paper; start out at 100% and then deduct points from there.

Generally what I do is start from an "average" baseline and nudge it from there by judging by what is strong or weak. It's just more natural for me that way. Nonetheless, I think I end up with a lot of fours as I don't really grade on a curve, but decide by suitability. The factors I typical consider are: value for content delivered, presentation, system fidelity, utility, and ideas.

The main problem with those that suggest we should be grading on a curve is that not all products get reviewed. I think a preponderance of reviews are about things that people actually buy, and people having learned to discern a bad buy before they buy it typically if you own a product, you are probably already looking at a high average.
Further, it's a moving target: products are getting better. Though I am still surprised that there are people who try to get by on the principle that fun is more important than usable statistics, in general people are learning what works and what doesn't, and I am seeing fewer and fewer dogs.

 

[tensen]

I forgot about that forum. I should go check it. And while I am at it, maybe I should peek in on that forum by our old pal RJR just for a lark...

Clark

That was cold man.
Thanks, it was getting too hot here, we needed someone airconditioning.

Seriously. The whole review process should be by the fans.
If you look over on mortality.net, there is a current group that is taking some feedback from the publishers and working on a review gorup that goes out and plays the material before giving a review. I think they are going to need a lot of extra support group, since their review time is going to be considerably longer... but it will add a great benefit to show how material stands up during play. I'm sure the reviews will also be greatly skewed from those reviews given to products just by a cursury view.

 

[AtFantasyReviewGuru]

Review standards

Excuse me, Big Brother . . . I find the idea of universal review guidelines incredibly oppressive. Sure, not all reviews can be well-done, but, what can I say, it's the Internet. Buyer beware, as always. Sure, we'll inevitably get badly-done reviews (there is one review site where I find the reviews consistently lacking in information about the product). In my opinion, reviews are mostly for getting information about the product so you can decide whether you want to buy it or not. My reviews are probably not as well-known as, say, Alan D. Kohler's, but I do generally get hundreds of hits on most d20 reviews (note: please, no one take an average of the number of hits of each of my d20 reviews and tell me I'm wrong; I'm just guessing). And at AtFantasy we don't even have a numeric system for reviewing (it is a bit misleading anyway)—I just say what I like about the product, and what I don't. I try to keep it short because I hate reviews that are too long (if I read them at all, it's at work). But mostly the opinion part takes a backseat—true, it's there, but the main purpose of reviews (why I read them, anyway) is to get what's really in the book (which is often not provided by publishers). For example, a publisher may say "30 prestige classes" (or whatever) in a product, but it's the reviewer's job to say what they are, or at least some of them. When there's a great number, however, that becomes prohibitive, so a few examples should suffice.

But as far as review standards go, that's probably just going to suck the life out of reviews, particularly coming from a publisher (Nat. 20 Press).

 

[pennywiz]

I'm hoping to see more paragraphs in reviews!

 

[Orcus]

I dont think people mean "standards" like some hard and fast set of rules. I think it is more like "guidelines," at least thats what I meant.

For example:

1. If you have a connection to the company whose product you are reviewing, state it.
2. If you recieved this product for free as a review copy, state it
3. If you playtested the product, state it.
4. If you are critical of the product, please give examples and full reasoning
5. If you have a bias against a genre, state it.
6. Provide examples of other products you can compare the product to. That helps the reader understand your take on the product and your biases as well.
For example, saying "the village in this product is stale and cliche." That doesnt tell me much. But if you said, "The village in this product is stale and cliche, reminding me of the Village of Homlet", that is useful for the reader. To someone who loved Homlet, they may now want to get the product, where before they might not.
7. Try to avoid the "holier than thou" tone (usually only present in poor reviews, but still worth noting)
8. Dont be condescending.
9. Try to find something good in the product. Even the worst product I have seen had something of value in it.
10. Tell us why you decided to review this particular product.

That sort of thing.

It isnt a list of commandments, it is just a bunch of things to keep in mind that frankly will help reviewers write a better review anyway.

Besides, why should reviewers be immune from reviews. Reviewers review our products and expect us to chage our stuff based on their reviews, why cant publishers review reviewers and suggest ways to chage reviews that are more helpful to us as publishers. No one is trying to be "oppressive," just helpful.

Clark

 

[Morrus]

Re: Review standards

Excuse me, Big Brother . . . I find the idea of universal review guidelines incredibly oppressive. Sure, not all reviews can be well-done, but, what can I say, it's the Internet. Buyer beware, as always. Sure, we'll inevitably get badly-done reviews (there is one review site where I find the reviews consistently lacking in information about the product). In my opinion, reviews are mostly for getting information about the product so you can decide whether you want to buy it or not. My reviews are probably not as well-known as, say, Alan D. Kohler's, but I do generally get hundreds of hits on most d20 reviews (note: please, no one take an average of the number of hits of each of my d20 reviews and tell me I'm wrong; I'm just guessing). And at AtFantasy we don't even have a numeric system for reviewing (it is a bit misleading anyway)—I just say what I like about the product, and what I don't. I try to keep it short because I hate reviews that are too long (if I read them at all, it's at work). But mostly the opinion part takes a backseat—true, it's there, but the main purpose of reviews (why I read them, anyway) is to get what's really in the book (which is often not provided by publishers). For example, a publisher may say "30 prestige classes" (or whatever) in a product, but it's the reviewer's job to say what they are, or at least some of them. When there's a great number, however, that becomes prohibitive, so a few examples should suffice.

But as far as review standards go, that's probably just going to suck the life out of reviews, particularly coming from a publisher (Nat. 20 Press).

I was referring to my own website. I have no particular interest in what you do at AtFantasy.

Plus what Clark said.