How would you like to see the reviews graded?

[Dragongirl]

There has been a lot of discussion about the merits of the current rating system used in the reviews area of the site. Please don't use this thread to continue those discussions there are other threads for that. This is simply to try to find out what you, the users of this site, would like to see.

Edit : Oh bother! Can a mod change the typo in the poll from "wan" to "want". Thanks.

 

[Dragongirl]

I voted for no ratings. By this I mean no stored and accumulated ratings by the site. This would not stop people from posting a rating within their review, but if there is no stored rating for a product then that would probably stop people from posting reviews just to boost a rating.

 

[Nightfall]

There's my vote.

 

[Crothian]

The rating system seems to be the fast way for people to get an idea of how a product is. It is too bad that people seem to want to abuse it to try to make certain products rate higher or lower for no other treason then they want to. Personally, I like the ratings, but I read the reviews. However, becasue of the abuses in the rating system, I voted no rating syastem. It seems like the easiest way to get rid of that.

 

[Chun-tzu]

I prefer a 10 point scale to a 5 point one, because for me, most products (the ones I buy, anyway) fall into the 7-10 range, and with a 5 point scale, it all seems to get a 4.

But we've already got a substantial database, and I don't know what changing the scale would mean (i.e., having 2 separate scales, or averaging, or asking reviewers to re-score products, or what).

You could also go with the standard American grading system, from an A+ to an F.

 

[Dragongirl]

You could also go with the standard American grading system, from an A+ to an F.

Darn, wish I thought of that when I made the poll.

 

[rounser]

Almost completely a suggestion of whimsy rather than being realistic:

There may be a call for "halo effect" and reviewer bias checkboxes like:
Reviewer admits to being a fan of the author/dislikes the author
Reviewer admits to having judgement significantly influenced by nostalgia
Reviewer admits to what they see as being a cool factor of concept of the product (e.g. "big, big dungeon!") significantly affecting their judgement of the actual execution of said product
Reviewer admits to what they see as being a poor concept of product affecting their judgement of the actual execution of said product.
Reviewer admits to judging product more harshly or more favourably because it's published by WotC/not published by WotC
Reviewer admits to being a fan of/disliker of cthulhu elements
Reviewer admits to being a fan of/disliker of psionics
Reviewer admits to being a fan of/disliker of the lower planes
Reviewer admits to being a fan of/disliker of low magic and "grim n gritty"
Reviewer admits to being a fan of/disliker of high magic and high fantasy
Reviewer admits to being a fan/disliker of settings X, Y and Z
Reviewer admits to preferring crunch over fluff
Reviewer admits to preferring fluff over crunch
Reviewer admits to only having done review to bump up the rating of said product
etc. etc.

That way you could match your own preferences and biases to those of the reviewer, and decide that because their priorities don't match yours, to take their words with a grain of salt or more. Reviewers don't like to think of themselves as being biased by stuff like nostalgia or being a fan of the author, and it's pretty easy to tell when they are, but some of them hide it pretty well and you end up buying something with dodgy execution based on their halo effected or "I'm a fanboy/hater of X so I'll flagwave for it/throw mud at it" reviews. Caveat emptor.

Altogether, that's way too many checkboxes. Perhaps reviews could have an "admitted reviewer bias" section instead. But then again, most of us are blind to our own biases, or see them as "being objective", so the above suggestions are probably pointless.

Perhaps the more biases a reviewer admits, the more their review is respected. It would cut down the amount of reading time of multiple reviews as part of "getting to know the reviewer" and trial and error in buying products they recommend to see how your preferences and biases match those of the reviewer.

 

[Khan the Warlord]

I voted for a simple "thumbs up or down" rating.

The reviewers should provide a clear, complete review and simply give a nod to if they believe one should purchase the product or not.

 

[MeepoTheMighty]

I would like to see a 1 to 10 rating on a number of different aspects of the product. For instance,


Fluff: 7/10
Crunch: 9/10
Art: 3/10
Layout: 9/10
Cartography: 3/10
Editing: 8/10
Value: 6/10


The "Fluff" would cover the concept behind the product, the writing style, and the overall feel evoked by the text. It would be a measure of how inspiring the plot devices are, or how solid the story behind an adventure is.

The "Crunch" section would focus on things like rules balance, proper statting of NPCs, and the mechanical represantations of the fluff. How well do the rules presented support a concept or idea? How consistent are they with accepted d20 conventions?

Art would obviously rate the interior and exterior art, both on its artistic merits and in relation to the subject matter.

Layout would cover things like use of margins, whitespace, fonts, and shading. How readable is the book? Does it make your head hurt just to look at it?

Cartography is self-explanatory for products which have maps.

Editing would rate the ability of the publishers to go over their texts with a fine-toothed comb, looking for rules discrepancies or obvious misspellings or omissions.

Value would be a gut reaction of "was it worth what I paid for it?"

These could either be averaged to find a composite score, or given as a total # of points. Maybe a few other categories could be added to make it a clean 100 point scale.

 

[Chun-tzu]

I had a similar thought, Meepo, but was thinking of the following categories:

Ideas 7/10
Mechanics 5/10
Presentation 9/10

Often, I'll find that supplements have some interesting ideas that might inspire me in some way, but there's something about the mechanics that bothers me.

Then again, mechanics can be tough to rate, since your first impression might be flawed.

Presentation collapses art, layout, cartography, and probably editing.