So why don't reviews work? (as a marketing tool)

[Scurvy_Platypus]

As a purchaser of products...

I pay attention to reviews. I explicitly seek them out, and I usually try for several different reviews.

Depending on the site, I pay attention to certain reviewers because they're pretty clear in what they like and don't like, and how the product in question matches up with that. The problem is that reviewers are people and therefore it's going to be hard to get a truly "objective" review. Some reviewers I have to read between the lines a bit to figure out if their perspective is going to be one I agree with or not.

For example, art.

I personally don't really care as much as everyone else seems to about art. Past a certain point, yes art can be a detriment. Otherwise, it's just there. Some people loathe the art style of D&D and its "dungeonpunk" or whatever deragatory term they're using these days, and other people love it. I don't care whether the reviewer loves it or not, I just want to know _what_ the art is, and if the art is a good example of that kind of _style_ or not.

RpgNow/DriveThru rpg, I find the reviews/comments to be mostly worthless. Some of the staff reviewers are fine, but I have yet to read anything left by a non-staff reviewer that I felt would actually influence my opinion one way or another. There really isn't much that can be done about this, as customers in general just aren't going to think about saying much about the product in question other than "I liked it" or "I hated it" and maybe a couple of other sentences.

As has been pointed out, reviews are pretty subjective. When someone buys something like Half-Life, they're buying a pretty specific product and game experience. In general, you're not going to buy an FPS game if you're not into that genre. RPGs on the other hand have all sorts of people buying them for all sorts of reasons. Some are collectors, some want to stripmine for ideas, some are into art, some treat the books like novels, some are completionists, some are fanboys, and then there's a few that actually seem to want to run the darn thing.

The problem is that with Reviewers, it's awfully hard to figure out _why_ they're reviewing a product in the first place. That's where knowing what the reviewer in question happens to like and dislike can be useful. It will often tell you whether or not the review is actually going to look at the product from a perspective that's useful to me the purchaser.

I've considered writing reviews myself of various products I've purchased, but haven't actually done so. Mainly because I feel that what I want out of a product is sufficiently different than apparently most gamers, so I don't see the point to putting in a lot of time trying to write a well crafted review that's just not going to be of use to most people.

 

[Arnwyn]

So I read review to understand what the book is about. I don't care much about the opinion of the reviewer because reviews are so subjective. The RPG hobby is very diverse. Even in Greyhawk, there's sub-division of those who prefered Gygax era and others that prefered 2E Greyhawk. Even with a new core setting of Greyhawk, some Greyhawk fans won't be happy with it. So my point is that I have taste very different from mainstream RPG. So I just want a summary of the product. Few or no opinion please.

Ah, so you don't like reviews. What you want is a table of contents - which you can get from a (competent) publisher's website.

Reviews are clearly not for you.

 

[jaerdaph]

I'll certainly agree with you on the unscrrpulous part. I've seen my fair share of that in the past, but I'm not sure there's a lot that can be done about that. Publishers can milk the system quite easily (on RPGNow and Enworld, though not so much on RPG.net), and it's ultimately up to the readers to discern what's what, and what's not.

I agree - there isn't a lot one can do about it except to become more educated as a consumer and learn to spot fraud. Caveat emptor and all. My political commentary self-edited out.

To be fair, as well, the system at RPGNow is currently written so that only people who have purchased a product can 'review it'. Publisher coupons are hence generally sent not to reviewers but to those who have purchased products.

It's not build into the RPGNow system, but I've received emails from several publishers that say if you leave x number of reviews for them, they will send you a discount coupon or gift certificate or put money in your account etc. So yes, the review will always come up as a "verified purchaser" review that way.

Here's another question, though: How would you improve it? Ban reviews shorter than a certain length like here on ENWorld? Get staff reviewers to verify each review written?

I'm not sure what can realistically be done. I'd like to see "comments" at RPGNow that consist of "kewl" or "sucks ass" or a string of gibberish characters just so they can rate something one to five removed. Perhaps there should be a minimum length requirement. While I personally don't see any value in the one to five rating at RPGNow anymore, I'd like to see the one to five rating someone leaves "weighted" somehow so that it counts more if the more comments you leave as well as for the more publishers you comment on. But I really don't think anyone is going to pay to have someone "screen" comments let alone volunteer to do the job. Staff reviewers certainly have enough on their plates (they tend to provide actual reviews, not comments), even if they do get "free" product, so I wouldn't want to see that task dumped on them.

But at the end of the day, I make up my own mind based on the recommendations of folks I know in message board discussions along with the participation of the designer or publisher. Comments and reviews, no matter the source, are always a secondary consideration for me, but they can hold weight if I'm sitting on the fence about a product's use for my games.

 

[Kaladhan]

Ah, so you don't like reviews. What you want is a table of contents - which you can get from a (competent) publisher's website.

Reviews are clearly not for you.

I want a little more than the table of contents, but basically, you're right.

I'm surprised no one mentionned this. When you talked about game reviews, you don't normally hear from who specifically it it. Rather, you learn that X magazine has given 4 stars to Y game.

The equivalent would be for EN World to give 4 stars to a RPG product. A website giving the notes might be more noteworthy than an individual. Everyone giving reviews kind of dillute (sp?) the whole thing.

 

[Voadam]

I want to know if the description screams plothook and inspires Storyteller.

. . .

Few or no opinion please.

Isn't that a matter of opinion?

I use reviews to gain info on the products as well. I usually buy off of rpgnow and cannot generally browse through more than the first four to six preview pages of a pdf. The product description is usually lifted off the paragraph blurb from the back cover of the product and is short.

Reviews often tell me things on a more in-depth level than from the product blurbs. So the fact that a review says a new d20 magic system in a product uses a spell point system with skill checks and fatigue or that it is more like a warlock variant style of mechanics tells me more than the "Includes an innovative new magic system for the d20 game!" blurb from the publisher's description.

I prefer sorcerer style spell casting to cleric style spellcasting so I look for that level of detail when considering shaman books. I usually find that in reviews and not publisher descriptions.

I dislike time travel so if a reviewer comments on its presence in describing a module's plot I know it is less to my tastes.

I dislike things changing PCs around completely so module reviews mentioning these types of scenarios "it requires the PCs to be captured, killed, then their brains are recovered and put into flesh golem bodies." are useful to me.

I generally could care less about the scores, it is the details of the product and why the reviewer feels certain things are good or bad that I find useful.

etc.

 

[R_kajdi]

There's nothing strange in that.
Most publishers publish what they consider good products. If a reviewer agrees, they're more like to get another review copy. If the reviewer then disagrees with the publisher on the next product it's easy for a publisher to accept a difference of opinion as there is at least some shared opinion. If the balance shifts towards negative, a publisher realizes that either the reviewer is a moron ()or just has preferences that aren't congruent to what they publish.

I think that's a good bit of why I find reviews to be not incredibly valuable. Why am I going to bother to read something that's basically been handpicked from among a bunch of fanboys to ensure a favorable rating? It's the same reason I don't bother with video game reviews. The only way to have a decent review process would be to go the Consumer Reports route, and that doesn't work in this case because of the long lead times in writing reviews. It takes me at least a week to digest the latest WotC length book, and even then the subtle issues only come up during play.

The other issue I've seen with reviews is that, especially with "big name" writers, there tends to be a huge fan effect to drown out the negative reviews. As an example, you need look no further than EN World's Return to the Temple of Elemental Evil reviews. A bunch of Monte Cook fans started spamming reviews at one point, mainly because they didn't want to see the module go to a low "average" review level.

As a consumer, if I bother to look at reviews anywhere for anything, I usually try to only read the negative ones. They tend to be the most honest about the product-- in a few cases, I've looked into getting stuff because the drawbacks that are sighted are not significant for me. Piles of glowing reviews by the usual biased sources don't interest me any more than normal marketing releases.

 

[Crothian]

I think that's a good bit of why I find reviews to be not incredibly valuable. Why am I going to bother to read something that's basically been handpicked from among a bunch of fanboys to ensure a favorable rating?

We're not all fanboys I can promise you that.

 

[Li Shenron]

I think that's a good bit of why I find reviews to be not incredibly valuable. Why am I going to bother to read something that's basically been handpicked from among a bunch of fanboys to ensure a favorable rating?

You can spot a fanboy's review from the first lines.

But there is some truth in what you say, that reviews are naturally shifted towards the positive rather than negative. The reason is just that people are more motivated to review what they like rather than what they dislike. In those few reviews I tried to write I think I was very picky about all the little flaws... but of course I gave the books a 4 or 5 at the end, because I'm very careful (read: detective-style) before I buy each book and so I end up buying only good ones I suppose I could try to borrow bad books from others just for the purpose of lowering my average, but really I don't feel very motivated in doing that...

 

[Nomad4life]

I tend to only read reviews when I'm trying to talk myself out of buying something. My first impulse is just to buy something that catches my interest on the spot- I have to seek out the reviews with the lowest ratings to convince myself I don't really need it after all.

 

[The Grumpy Celt]

I will read reviews to check things out, so they do guide my purchases. My editor says they help.