Redesigning the Reviews Page
trancejeremy
Adventurer
It would be nice to give half a score, like 3.5 instead of having to pick either a 3 or 4.
I agree about 7 being the minimum is too big. Not many non-WOTC products have that many, with the exception of some of Green Ronin's stuff, and SSS/Necromancer.
I would also be nice to be able to add a product to the review database. I mean, it seems to be missing a lot, and with more being added all the time, it would be quite a job for someone to keep it up to date. Especially for stuff that won't get reviews, or only 1.
It might be best to restrict it to someone who has written 10 reviews (or so many kb worth), so it's not abused. So no one adds something like the Quintessential Pimp.
As for a weighting system, maybe you could let people vote on it. I always trust Psion's reviews, for instance, so I would give them a lot of weight. But some others, well, I don't give them much, if any.
I agree about 7 being the minimum is too big. Not many non-WOTC products have that many, with the exception of some of Green Ronin's stuff, and SSS/Necromancer.
I would also be nice to be able to add a product to the review database. I mean, it seems to be missing a lot, and with more being added all the time, it would be quite a job for someone to keep it up to date. Especially for stuff that won't get reviews, or only 1.
It might be best to restrict it to someone who has written 10 reviews (or so many kb worth), so it's not abused. So no one adds something like the Quintessential Pimp.
As for a weighting system, maybe you could let people vote on it. I always trust Psion's reviews, for instance, so I would give them a lot of weight. But some others, well, I don't give them much, if any.
Monte At Home
Explorer
Top Ten Lists
I think that you should either adopt a system like CRGreathouse
suggests, or you should drop the top and bottom 10 lists altogether. Currently, they are deceptive because they don't include all the products. Before, they were deceptive because people could (and by appearances, were) spiking ratings with good or bad reviews just to alter the top 10 lists.
I wouldn't weight reviewers differently I don't think. I certainly wouldn't base it on number of reviews done--it encourages people do a lot of reviews, but not necessarily good reviews.
You could (although this would be complex) have people rate how useful they felt various reviews were, and then weight reviews based on previous reviewer ratings, but I don't think that's necessary.
Mostly, I think that if you're going to have a site where anyone can come along and review any product, you should treat all the reviewers the same.
I like very much that the recent reviews are listed, and that you can click on publishers.
I think that you should either adopt a system like CRGreathouse
suggests, or you should drop the top and bottom 10 lists altogether. Currently, they are deceptive because they don't include all the products. Before, they were deceptive because people could (and by appearances, were) spiking ratings with good or bad reviews just to alter the top 10 lists.
I wouldn't weight reviewers differently I don't think. I certainly wouldn't base it on number of reviews done--it encourages people do a lot of reviews, but not necessarily good reviews.
You could (although this would be complex) have people rate how useful they felt various reviews were, and then weight reviews based on previous reviewer ratings, but I don't think that's necessary.
Mostly, I think that if you're going to have a site where anyone can come along and review any product, you should treat all the reviewers the same.
I like very much that the recent reviews are listed, and that you can click on publishers.
trancejeremy said:
Agreed.
I shudder to think what we'd end up with. ..
"players handBook" by "wizards", "A bok for plying dnd". "Oh, sorry, I didn't realise what you meant by an ISBN number, I thought I was supposed to enter my favourite page number there."
EricNoah
Adventurer
I have a thought or two about the d20 guide...
I'd like to see colorcoding or some other visual aid to indicate when an item is...
available
not yet published but has a set release date
cancelled
out of print
is not available and has no set release date
comes out periodically (like a magazine, etc.)
Each publisher entry should have what's already in the guide (publisher name, general publisher/product description, website, fan sites, a picture of an upcoming or well-known product from that company, and lists of products with release dates.
However what's not really codified quite yet is a system of marking products by different types (adventure, adventure anthology, rulebook, campaign setting, campaign accessory, monster book, stand-alone d20 game, etc.) -- people will want to search by those types -- and products should be able to be put into multiple categories. The product format (printed softcover, printed hardcover, download, packaged software, etc.) should also be indicated. And there should be a place to link to the publisher's webpage for each product. And then that's also where there could be links to the D20 Reviews for each product.
Each product might also be part of a "product line" like Penumbra or Freeport, etc.
We don't have much author info for products listed currently, that could be another thing people might want to search for (all of Monte Cook's products across different publishers).
And finally the review system should include not only reviews by us folks but a way to add links to other websites' reviews.
And maybe a very unobtrusive link to where the product can be purchased at EN World's online store.
I'd like to see colorcoding or some other visual aid to indicate when an item is...
available
not yet published but has a set release date
cancelled
out of print
is not available and has no set release date
comes out periodically (like a magazine, etc.)
Each publisher entry should have what's already in the guide (publisher name, general publisher/product description, website, fan sites, a picture of an upcoming or well-known product from that company, and lists of products with release dates.
However what's not really codified quite yet is a system of marking products by different types (adventure, adventure anthology, rulebook, campaign setting, campaign accessory, monster book, stand-alone d20 game, etc.) -- people will want to search by those types -- and products should be able to be put into multiple categories. The product format (printed softcover, printed hardcover, download, packaged software, etc.) should also be indicated. And there should be a place to link to the publisher's webpage for each product. And then that's also where there could be links to the D20 Reviews for each product.
Each product might also be part of a "product line" like Penumbra or Freeport, etc.
We don't have much author info for products listed currently, that could be another thing people might want to search for (all of Monte Cook's products across different publishers).
And finally the review system should include not only reviews by us folks but a way to add links to other websites' reviews.
And maybe a very unobtrusive link to where the product can be purchased at EN World's online store.
Good feedback
So there are some great ideas. And some not so great ideas, but thats part of the process.
Some things I think make sense so far:
I will set up a few layers of permissions:
1 - Unregistered User - Can only read reviews
2 - Registered User - Can read and write reviews, plus comment on products
3 - Asst. Administrator - Can submit products for the product list
(but must be approved by an administrator)
4 - Administrator / Editor - Full rights to system
That way we can manually designate those who can add products to the list, but they won't actually appear in the list until they are approved by Morrus (or whoever else he designates as an Administrator)
I also like the idea of categorizing the products, like supplement, periodical, core rulebook, module, etc.
I'll have to think about a way of handling periodicals, it would be cool to be able to group, say, all Dungeon magazines into one area, so that we could have issue by issue reviews and an overall rating for the periodical.
I think clearly the front page of reviews needs to be simplified. Listing top lists in various categories, the newest reviews, newest products added, and maybe top reviews would be good, and then you could select any category to get a full listing.
Perhaps the top ten lists could be split into two parts: Products with over X amount of reviews, and products with less than X amount of reviews.
I also plan to add some kind of Alphabet pick list for the different categories, and a way of browsing through products.
A simple text search will also be added (likely to be very simple, for example search by name 'handbook' would get back all products with the word handbook appearing in the title.
Another feature might be if you select a reviewer, not only can you get back all their reviews, but you could get back a list of all comments for the different products.
Understand that most of this is not difficult in itself, but we may phase features in piece by piece, as I can't do everything at once.
Anyway, I'm excited by all the feedback. Keep it coming.
Erithtotl/Ian
So there are some great ideas. And some not so great ideas, but thats part of the process.
Some things I think make sense so far:
I will set up a few layers of permissions:
1 - Unregistered User - Can only read reviews
2 - Registered User - Can read and write reviews, plus comment on products
3 - Asst. Administrator - Can submit products for the product list
(but must be approved by an administrator)
4 - Administrator / Editor - Full rights to system
That way we can manually designate those who can add products to the list, but they won't actually appear in the list until they are approved by Morrus (or whoever else he designates as an Administrator)
I also like the idea of categorizing the products, like supplement, periodical, core rulebook, module, etc.
I'll have to think about a way of handling periodicals, it would be cool to be able to group, say, all Dungeon magazines into one area, so that we could have issue by issue reviews and an overall rating for the periodical.
I think clearly the front page of reviews needs to be simplified. Listing top lists in various categories, the newest reviews, newest products added, and maybe top reviews would be good, and then you could select any category to get a full listing.
Perhaps the top ten lists could be split into two parts: Products with over X amount of reviews, and products with less than X amount of reviews.
I also plan to add some kind of Alphabet pick list for the different categories, and a way of browsing through products.
A simple text search will also be added (likely to be very simple, for example search by name 'handbook' would get back all products with the word handbook appearing in the title.
Another feature might be if you select a reviewer, not only can you get back all their reviews, but you could get back a list of all comments for the different products.
Understand that most of this is not difficult in itself, but we may phase features in piece by piece, as I can't do everything at once.
Anyway, I'm excited by all the feedback. Keep it coming.
Erithtotl/Ian
Psion
Adventurer
Making my case
If you had seen what my eyes have seen...
Ahem... I really think I need to make my case here. I think that if you don't do something like weight the reviews, the average system suffers.
There have been times that people have posted a review and someone has decided that they disagree with the ratings and are sign up just so they can contradict the review they don't like, and they assign it the most extreme score that they can (a 1 or a 5). This compromises the fairness of the system, because the "reactive" score give is not out of an honest assessment of what they think the product is worth, but an attempt to annul the review that they didn't agree with and to bump the average in the direction they want to. I have seen this multiple times.
I understand what you are getting at by saying people can write more reviews but not good reviews. However, that is why I suggested using the root of the number of reviews as a baseline. You get little payoff by just writing 2 or 3 extra reviews just to get your point across about one, so as a result it would discourage this sort of reactive behavior... or at least lessen its effects. If more products got more reviews, it might not matter, since it would all average out. But there are really not that many reviews on a lot of products, so I think some artifical compensation is warranted.
Okay, on another note... I recommened earlier that you could half the weight of the highest and lowest review. I think Charles' idea of using the root(mean*median) acheives much of the same thing but would be easier to program since you don't have to worry about the sticky issue of how it related to the weighting system I recommended above.
I think number of reviews is a simpler metric, and will be sufficient to address the syndrome that I am worried about.
Monte At Home said:
If you had seen what my eyes have seen...
Ahem... I really think I need to make my case here. I think that if you don't do something like weight the reviews, the average system suffers.
There have been times that people have posted a review and someone has decided that they disagree with the ratings and are sign up just so they can contradict the review they don't like, and they assign it the most extreme score that they can (a 1 or a 5). This compromises the fairness of the system, because the "reactive" score give is not out of an honest assessment of what they think the product is worth, but an attempt to annul the review that they didn't agree with and to bump the average in the direction they want to. I have seen this multiple times.
I understand what you are getting at by saying people can write more reviews but not good reviews. However, that is why I suggested using the root of the number of reviews as a baseline. You get little payoff by just writing 2 or 3 extra reviews just to get your point across about one, so as a result it would discourage this sort of reactive behavior... or at least lessen its effects. If more products got more reviews, it might not matter, since it would all average out. But there are really not that many reviews on a lot of products, so I think some artifical compensation is warranted.
Okay, on another note... I recommened earlier that you could half the weight of the highest and lowest review. I think Charles' idea of using the root(mean*median) acheives much of the same thing but would be easier to program since you don't have to worry about the sticky issue of how it related to the weighting system I recommended above.
I think number of reviews is a simpler metric, and will be sufficient to address the syndrome that I am worried about.
Similar Threads
