What constitutes a good vs bad review?

[Psion]

First off, it seems some posters here have skipped the tracks. From the first post, it seems that Utrecht mean "positive or negative" reviews.

AFAIAC,
4+ is a positive review.
3 is a neutral review.
2- is a negative review.

(To be more explicit:
1: This product is useless even if you are interested in the subject matter. To date, I have only ever awarded one "1" at ENWorld.
2: The product has some salvagable material, but has major weaknesses.
3: The product has strengths and weaknesses. Might be worth picking up if you are greatly interested in the subject matter.
4: The product is strong and makes me want to use it in a game. You should pick this product up if you are interested in the subject matter.
5: The product is imaginative and well executed. I would actively strive for ways to use this product in my game.)

In general, a 3/5 is a bad review, IMO.

I intensely dislike this viewpoint, and fight it at every turn. Why? Because if you limit me to giving products a 4 or 5 unless I actively dislike it, then you are not giving me enough room to be discerning.

I do find this attitude is pretty prevalent, however. I frequently find that awarding a publisher a 3 on a review will result in hostility as the publisher thinks you are "out to get him." This has happened with no less than five publishers with me that come to mind. Which I think shows a certain disdain for one's peers. Well sorry, but every work can't be "above average", and when I give a product a 3 that means that it is "satisfactory", something you might want to consider buying.

I think many reviewers who give out only 4s and 5s are entirely too generous. A 5 should be reserved for something really special, and a product that has some significant drawbacks along with some strengths should have a 3 not a 4, IMO.

 

[MEG Hal]

Well, I try to grade on a curve, so to speak, so I personally consider a 3 out of 5 to be average. I also consider the "average" quality of most RPG books to be pretty good. Worth buying, at least (though maybe not at full price)

Many publishers seem to be offended when they get anything less than an A or a 5. But I try to save those for great products.

(I do agree that a good review will tell enough about the book to let the reader make up their own mind, but that can be tricky, because some books, like say, monster books, are hard to describe accurately without going into an insane amount of detail)

I agree with paragraph 1. What I look for is the number first and 3 is average, then a write up about the product. What I look at is did the reviewer read the book in question and understand it, I would rather have a 1 with details about the book and explaining why they gave it a 1 then a 4 stating "it was good". A review should explain what the book is and why they thought it got the grade it did.

And I agree that a 5 should be for the cream of the crop. so from a publisher's view what I like is a well done and explained review, regardless of the number assigned.

I think I am in the norm of publishers not the odd man out.

*An example is a reviewer stated he did not like that our book Monte Cook's Arcana Unearthed: Siege on Ebonring Keep did not convert the info to d20 or have info on X monster when it was clearly stated on the cover it is an OGL book and requires Monte Cook's Arcana Unearthed so we got marked down for that, that gets a bit frustrating, but again if you read the book and explained the book with understanding I am OK with any score given. The fact that we got lower marks because we did not do something that the reviewer should know was not a part of the book is just plain unusual(IMHO).

 

[Psion]

One thing I should add:

I try to make my analyses in the review as objective as possible, and explain why I came to the conclusions I did about the product, so if you don't find something that I didn't like to be a problem, then you can make that determination yourself.

However, there is no way that I can reflect that level of sensitivity in the final score. What is a 5 to me may be a 3 (or less) to someone else. It should be understood that my final tabulated score is primarily a reflection of my weighting criteria. If you have similar weighting criteria, you will probably find that what I give a high number rating is a good product for you, but if you don't find some things as important as I do, your mileage may vary.

 

[frankthedm]

3/5 means the product has average level quality to the reviewer.

 

[NewJeffCT]

I think it depends completely on the reviewer. If a reviewer has a body of work and what he or she has reviewed averages out to a 3 out of 5, I'd say that 3 is average, 4 is good, 5 is excellent. However, if a reviewer has an average product rating of 4.5, does that mean if he or she rates something a 3, it is terrible?

But, you also have to take into accounts your own tastes and the tastes of the reviewer...Some people like their d20 books to be rules heavy with lots of new prestige classes, spells, monsters, magic items, alternative rules, etc. They might not like a 'rules-light' supplement. However, a person that likes a lot of backstory and history, or a plot heavy module with a lot of political intrigue and complex NPCs, might love the same 'rules-light' book. Personally, I prefer the rules-light supplements and am always looking for fresh, interesting ideas. But, I know a lot of people prefer the rules-heavy stuff.

 

[TiQuinn]

I intensely dislike this viewpoint, and fight it at every turn. Why? Because if you limit me to giving products a 4 or 5 unless I actively dislike it, then you are not giving me enough room to be discerning.

No, I want you to be discerning.

But I can only use so many accessories in my game, and I only have so much money that I want to spend on gaming stuff. But if a reviewer whose opinion I trust only gives it a middle of the road rating, I'm a lot less apt to pick it up. From my POV, I would call that a negative rating. I don't expect others to be as discerning, and I could always be missing out on something good. Part of it is there are some reviewers who give nearly everything a 4/5. Sometimes, I think reviewers are not discerning enough. I like high standards. I'll say this: when reviewers give something a 5/5 at ENWorld, I usually end up agreeing. 4/5 can be hit and miss, but it's usually a hit. 3/5 is a crap shoot. Unless I'm really interested in the material, I'll probably pass unless I can actually sit down and read it a bit before buying.

In the end, I always read the reviews, and I stick to reviewers who seem to dig the same things I do, so it's not like I'm basing my entire decision on just the score.

 

[Graf]

Psion: I think most people are discriminating. I've frequently purchased books you've given mixed comments on if I felt they weren't applicable to my game/gaming mindset.

Broadly I think that any review that gives you a good idea about
1. What's in the book.
2. What the reviewer liked about the book and why it appealed to them.
3. What the reviewer disliked about the book and why it didn't work out.
4. An overview of the mechanical soundness of any rules, with an emphasis on trouble spots and ideally, a discussion of why something looks like it cause trouble.

*An example is a reviewer stated he did not like that our book Monte Cook's Arcana Unearthed: Siege on Ebonring Keep did not convert the info to d20 or have info on X monster when it was clearly stated on the cover it is an OGL book and requires Monte Cook's Arcana Unearthed so we got marked down for that, that gets a bit frustrating, but again if you read the book and explained the book with understanding I am OK with any score given. The fact that we got lower marks because we did not do something that the reviewer should know was not a part of the book is just plain unusual(IMHO).

Not having read the book or the review I can't really comment on the review, per se. But I find the complaint contradictory.

This is a D&D/D20 site. It may be a great game system but Ars Magica stuff isn't covered on EnWorld. If an Ars Magica product came out and was presented as a supplement that might be appropriate for a D&D player/DM to look at I would expect a reviewer to discuss how much was useful to D&D, how much could be changed to be useful for D&D etc.

I expect the same for setting based products (is the material useful outside of the setting, does it require a lot of assumptions that aren't present in generic settings?). Likewise if a 3.0 product came out now I would appreciate comments from a reviewer about whether it incorporated 3.5 changes and how much effort would be required to change a book to match up with 3.5.

UA seems particularly likely to get into this problem due to its peculiar marketing: on one hand it's implied that its D&D compatible, covered on EnWorld, etc. On the other hand it's really its own game system. Most UA material requires at least some work to translate to D&D. A review to D&D oriented consumers mentioning clearing mentioning that aspect of the product, and considering that aspect in giving a review rating seems more than fair.

I mean, UA could, like so many other fantasy games that are similar-to-but-different-than-D&D, not be covered on EnWorld at all.

 

[S'mon]

2 is poor, 3 is average, 4 is good.

 

[Maggan]

Where is the middle?

I have another question related to this.

What is considered the middle level of quality? A level that the reviewer thinks is "this is what most people should be able to produce" or is it in relation to everything else out there?

Maybe I'm being confusing, but what I trying to figure out is in general are some publishers held to a higher standard than others (eg WotC, SSS, Green Ronin, Atlas) just because they consistently produce a certain level of quality?

I'm certain many individual reviewers hold everyone to the same standards, but in general I'm not that sure. Me for one (even though I'm not a reviewer) am more forgiving to WotC and SSS because I generally like their products. Not so with other publishers.

So, I guess my question is: how do we set the middle level as regards to quality?

Cheers

Maggan

 

[S'mon]

So, I guess my question is: how do we set the middle level as regards to quality?

Middle (expected) level is 3.

I judge WoTC products same as others - eg reviewing the 3.0 Monster Manual it was good value (got it for $20) with expensive flashy presentation, but over-cramped layout, art slick but not very evocative, and a dismal lack of both encounter tables and any entry for Men/Human. Since the latter 2 are important to me it got a 3, if it had had comprehensive encounter tables and a good treatment of humans-as-monsters I might well have given it a 5.