Re. EN World staff reviewers and good/bad reviews

[Wandering Bard]

Wow! I just found this thread (I've not been able to keep up to date on the boards...too busy). This is very interesting.

First off, Bard's Productions is not the company in question. However, I can understand how someone would get upset about poor reviews. In September-December when we were releasing the Race for Retribution PDFs, we sent 4 emails over a month's time to Psion offering him copies to review. We did not receive a reply to any of the emails. I assumed he was busy, or didn't do PDF reviews, or something like that, but a reply to state that would have been nice. So we contacted Simon Collins. Part I we got a 3, Part II a 2, Part IV and V were both 4's.

Needless to say, as the author, I was highly disappointed at the 2. I sent a private email to Simon for more explanation to some of his comments and received some interesting feedback which has proven very good. However, the problem is how certain comments were made in the review, such as the 3 way fight being done better in other products. When I asked for examples our email conversation (to learn on how to improve our product) he was unable to provide any solid ones (Simon guessed at one product but it turned out it didn't have a 3 way fight...I bought it to read through and learn from).

Simon also made a comment that he could draw better artwork than what was present in the PDF. I'll be the first to admit that some of our artwork was bad in the PDFs. We have been doing some serious updating on Layout, Maps as well as artwork. It should be noted that some publishers may have taken Simon's comment as an offer to re-do the artwork (i.e. put your money where your mouth is). Again, he made his point which was accurate on some of the artwork, but how it was made is questionable.

Now, in Simon's defense, these were some of his early reviews and so, he was still probably trying to define his style and polish it a bit. Alan, clearly is a solid reviewer, but as stated earlier, we never received ANY reply to the emails we sent (as our initial email to Simon indicated). Again, these are not major points and probably have been cleared up since our efforts in Sept - Dec. All in all, our email conversation with Simon was very informative and I appreciate his time with that.

We are going to be printing out some new products, Common Ground I (Churches, Inns and Merchants) as well as a completely updated Race for Retribution series. That's right...print...not PDF. We will be sending Simon and Alan copies of these products for them to review. Obviously we hope for 5's, but we also know there will be things that can be improved. There's nothing wrong with criticism, as long as it is constructive and more details concerning it can be given, to publishers who wish to learn from these observations.

Both Simon and Alan have done a solid (4 out of 5... ) job on reviews, even if on occasion I don't agree with their opinions or observations or weighing of certain factors (concerning other publishers material...not BP's). Keep up the good work and I'll be sending you guys emails on the printed product in a few weeks.

Sincerely,

Jeff Colledge
Bard's Productions, LLC

 

[Midnight Rider]

There, that's off my chest. There are a lot of great publishers out there who take the bad (or, more often than not, merely average) reviews in stride; that, in my opinion, is the professional way of acting.

Yes, well, you would think this. ENworld continues to become a less interesting place, less supported by a breadth of publishers, and is running straight downhill for anyone with an opinion. Fanboy raving is acceptable in the message area, but not strong opnions. Yet ironically, sharp critiques with poor reasoning and a lack of understanding or breadth of gaming knowledge, are allowed in the review area, where strict opinion is given an air of more meaning.

ENWorld is a snotty little world of mixed up values. It has no consistent policies except to protect its own ass.

 

[Midnight Rider]

I'll never understand this kind of behavior either. Having been the "target" of more than a few negative reviews, I know that getting them does piss me off, but it also gets me to hunker down and do all I can do to make the next product even better.

This eliminates the possibility of better products because anything which is creative, but ignores the current trend, will get a bad review, even if it is a really good product. If a journalist were reviewing a product which goes against the typical crap that is out there, they'd recognize what is going on with the product. But a fanboy dweeb who has never really given consideration to design, and who has merely copied the expected and already published materials, that same product will be harshly wracked.

If we want more powerful works than just the typically conformist drivel, then we need to take a different tack with reviews. Otherwise, the common denominator will become the standard, and the same old crap will bore the heck out of everyone. Quiet a few professionals in the industry concur. For instance, Ryan Dancey has said that 95% of the stuff out there is crap.

 

[Lizard]

Re: Re: Re. EN World staff reviewers and good/bad reviews

ENWorld is a snotty little world of mixed up values. It has no consistent policies except to protect its own ass.

Uhm...protect its ass from WHAT, precisely?

And why do I sense the distinct odor of sour grapes oozing from Mr. Rider's post?

 

[Glyfair]

This eliminates the possibility of better products because anything which is creative, but ignores the current trend, will get a bad review, even if it is a really good product. If a journalist were reviewing a product which goes against the typical crap that is out there, they'd recognize what is going on with the product. But a fanboy dweeb who has never really given consideration to design, and who has merely copied the expected and already published materials, that same product will be harshly wracked.

If we want more powerful works than just the typically conformist drivel, then we need to take a different tack with reviews. Otherwise, the common denominator will become the standard, and the same old crap will bore the heck out of everyone. Quiet a few professionals in the industry concur. For instance, Ryan Dancey has said that 95% of the stuff out there is crap.

There will always be room for innovation, and such a product will get good comments and bad comments just because it's being innovative. You seem to be the type of person who would raise a review you were giving one or two points just because it is being innovative...regardless of whether it's a good innovation or not. Even if you aren't, there certainly are others who are.

There are also traditionalists who will trash anything that isn't what they expected. An "innovative" product will get trashed almost regardless of it's actual value because what they are looking for isn't there.

As for the 95% of the stuff is crap, that's certainly subjective. However, while 95% of the usual products are crap, so are 95% of the "innovative" products. Most of such products seem to be put out by people who want to be different. However, in their search for innovation, they forget to make the product useful or appetizing.

There is also the handicap that innovative products, not having as much to build on as a traditional product, have major flaws in them. One of my favorite RPG systems is Hero Wars. I absolutely love the concepts behind the system and the character generation system is the best I've ever seen. However, there are major flaws in the system that deter most of my players from it. If I were reviewing it I couldn't give it a very solid rating. <As an aside, I'm hoping Heroquest is going to fix most of those flaws>.

Glyfair of Glamis

 

[coyote6]

This eliminates the possibility of better products because anything which is creative, but ignores the current trend, will get a bad review, even if it is a really good product.

How does that follow? I don't see the logic that makes it inevitable.

For instance, Ryan Dancey has said that 95% of the stuff out there is crap.

He probably said 90%; that's Sturgeon's Law. Ninety percent of everything is crap.

Including, I suppose, Sturgeon's Law.

 

[Midnight Rider]

Re: Re: Re: Re. EN World staff reviewers and good/bad reviews

Uhm...protect its ass from WHAT, precisely?

And why do I sense the distinct odor of sour grapes oozing from Mr. Rider's post?

I'm sorry, but we're not allowed to talk about moderators in ENworld. Therefore, if you don't see it yourself, it doesn't exist. and you can ignore that part of my post because it refers to something which doesn't really exist. Technically, my post which kind of criticizes ENworld doesn't exist either. I mean, if a moderator read this they'd probably tell me to not talk about it, and constrict me from criticizing them, so this post may not even exist by tomorrow morning. In fact, let me retract this post right now!

I didn't say they protect their own butts! They don't. Not really. At least, nothing seems to exist which would serve as evidence for this.

 

[Morrus]

Re: Re: Re. EN World staff reviewers and good/bad reviews

Yes, well, you would think this. ENworld continues to become a less interesting place, less supported by a breadth of publishers, and is running straight downhill for anyone with an opinion. Fanboy raving is acceptable in the message area, but not strong opnions. Yet ironically, sharp critiques with poor reasoning and a lack of understanding or breadth of gaming knowledge, are allowed in the review area, where strict opinion is given an air of more meaning.

In other words... you disagree with a review? Are you a publisher under an alt. ID? I can't think why anybody else would get so angry about a review.

If you have a specific problem with me, my email address is available. Feel free to make use of it.

 

[Turjan]

Well, this may be out of place if I consider the other posts in this thread, but I'm actually quite content with the reviews on this site. The reason for this may be that I actually judge each review on its own merits.

Why I like the reviews as they are...

I disregard reviews which don't give away any information about the contents, because I don't know whether the reviewer actually read the module.

I disregard reviews given by players who actually didn't read the modules, because I don't know how much time the DM spent in order to get his players into such an enthusiastic or murderous mood .

I don't have any problems with reviews by those people whose repertoire of ratings consists of either 5 or 2 points (depending on their mood ), as long as they give a detailed description of the product. This lets me judge myself whether I like those details or not.

I like those reviews best which tell me exactly on which details a reviewer based his final judgement, and how important these details were for his judgement. If the result is mainly based on price and artwork, I can do my own adjustments to the rating.

Any suggestions though?

Well, I think the mixture of staff reviews and fan reviews is a good one, anyway. This means that even those products may have a chance to get some coverage which otherwise wouldn't get any mention at all, due to time restrictions. But...I think, the readers (me included) should make more use of the "commentary" function. This is a good place to air one's concerns.

The rating system needs some tweaking though. All these thoughts about median or mean, which were made above, don't lead to anything productive. If you simply look at the worst 10 products, you will see, that some of them still get ratings higher than 3 (which should be above average ), while the list of the best 10 still contains products with ratings below 4. I don't honestly think a median would change this general picture.

Fact is, you won't get many fan reviews of bad products. Bad products are simply not bought in the first place. This leaves the actual differentiation scale shrunk to the range of 3 to 5. Therefore, I'd suggest to broaden the range from 0 to 10, or, if you don't want to lose the compatibility to the old system with only 5 points, allow digits; either just 0.5 steps or even more detailed. This would bring back some differentiation to the now more or less general rating of 3.8, with a few exceptions .

Turjan

 

[Psion]

The rating system needs some tweaking though. All these thoughts about median or mean, which were made above, don't lead to anything productive. If you simply look at the worst 10 products, you will see, that some of them still get ratings higher than 3 (which should be above average ), while the list of the best 10 still contains products with ratings below 4. I don't honestly think a median would change this general picture.

That is not the main intent of median weighting. Median weighting is not an effort to fan out the scores. Rather, it is an effort to minimize the impact of the all too common "emotive overcompensator." You know the type: a product comes along that some people have given 4's or 5's to. Then one person comes along and decides that it is only worth a "three" but since the average doesn't reflect what he thinks, he gives it a one instead.

Fact is, you won't get many fan reviews of bad products.

This is somewhat true. However, those one or two fans who took the hit for the team and bought some products like the Foundation did everyone else a service when they did review them and gave it the score it deserved.