Upgrade your account to a Community Supporter account and remove most of the site ads.

Reply to thread

Message: <blockquote data-quote="AaronLoeb" data-source="post: 1723460" data-attributes="member: 4382">I did not mean to imply or say that not much voting theory went into the system. I assume thought went into it, since thought has obviously gone into every other part of the ENnies. I believe there are alternative systems that would better achieve the goals this system was put in place to achieve. Repeated: I think the ENnies have been very well planned and very well executed and I think you guys have been very smart about them at every step. So please do not read my comments as derission for your considerable and impressive work.I agree wholeheartedly. OPOV is a lousy system. I only think it is less gameable than the current system -- a theory on my part that may be wrong.I can't speak to that as I have not seen the voting results. My comments are directed at a potential for abuse that may have never happened. Without looking at voting results, I can't argue this point about past results -- I'm addressing my comments to a potential problem. As the awards gain in popularity and stature, I think a new voting system would be a good idea.It is an excellent goal and a worthy example. The problems with the comparison are: 1) that the IMDB's rating system is meant to find a ranking from 10s of thousands of users on a single item not in comparison to others 2) it is not a contest, so there is less incentive to game the system short of just raw, fanatical prejudice (which the Internet is filled with), and 3) if they were using an "average" system on the rankings, on more obscure movies you would see stranger results. This is why IMDB uses a weighting method rather than a raw average -- to avoid the exact kind of "vote stuffing" I'm worried about. They don't reveal how they weight votes so as to keep people from gaming the system. But were they using just an average or mean, as I think the ENnies are (please correct me if I'm wrong), here would be a good example of the possible problem I'm referring to:Forbidden Zone is an obscure, cult-classic movie. Most people I've tried showing it to find it unwatchable. Most people haven't heard of it. Those who have are generally crazy for it.Here you can see its raw votes on IMDB:<a href="http://www.imdb.com/title/tt0080752/ratings" target="_blank">http://www.imdb.com/title/tt0080752/ratings</a>Because so many people have given this cult movie (that the average consumer, in my experience, finds unwatchable) a 10, its "average" score would be a 8.7. The weighted score is a 6.5. Again, I don't know how that weighting works. But let's go with this example. Only 587 people voted because most people haven't heard of it. And the overwhelming majority of those people were huge fans of this cult hit, so it got a mean score of 8.7.Now, let's look at Fellowship of the Ring. I don't really know what anyone else's measure of quality for a movie would be, but I think the majority of us who have seen Forbidden Zone and Fellowship would agree that Fellowship is a better movie. I know, everything here is subjective, so let's leave quality out of this. Let's just think of Forbidden Zone as a movie with a small, fanatical following and Fellowship as an incredibly huge movie everyone has heard of.<a href="http://www.imdb.com/title/tt0120737/ratings" target="_blank">http://www.imdb.com/title/tt0120737/ratings</a>134,363 people voted for fellowship. Over half of those gave it a 10 out of 10, but 6,344 of them gave it a 1 out of 10. Some portion of the voters wanted to pull down the score. Because of this, Fellowship's mean score is 8.6; lower than Forbidden Zone.In this example, we can see a tiny, fanatical minority of people earning a movie no one else has heard of a higher mean score than a movie everyone has heard of and some people wanted to pull down (the dark side of popularity). Were you to go for straight means, the voting scheme has benefitted Forbidden Zone for being obscure.It is clear that they do, and unclear what they are. I think a simpler system of assigning points to products would be superior. Their mathematical tricks are necessary because the 1 to 10 ranking essentially allows people to give a product not only out-of-the-average positive votes but also out-of-the-average negative votes. Any system that allows people to vote "negatively" is likely to prove unfair as it will benefit people who dislike a company on principle rather than having any actual opinion about the product.My assumption is that the average EN World member -- likely the overwhelming majority of members -- vote 7s to 9s on products. I am a prolific lurker on these boards and find the people here to be reasonable and fair minded. It is because of this that I think the voting scheme is flawed because it takes the power out of their hands. I assume you don't get a 1000 people voting 10s and 1s on the same products. I assume you get 1000 people voting fairly average scores and then a few hundred voting 10s for their favorites and 1s - 3s for the competitors, or 1s for the companies they hate regardless of the quality of the product. This small minority can unfairly tank a product (by voting 1s) that the overwhelming majority of voters find to be pretty darn good or can elevate a product that the overwhelming majority of users have never heard of (by giving it 10s that are not offset by very many "normal" votes).To use the Forbidden Zone example, if you have a product that most of the EN World community has never heard of (again, ignoring quality arguments -- maybe it deserves to win; that's not the point), it stands a very good chance of winning if it has a fanatical following. Why? Because the fair-minded people decline to give it any score. Then the fanatical minority comes in and gives it 10s. It then has a higher average than the products that most of the people who voted have heard of because they were all giving it "fair-minded" scores of 7 to 9.Similarly, if you have a magnificent product produced by a company that a vocal minority loathes (in the case of Fellowship, above, because they didn't have Tom Bombadil or something), it can be taken out of the running by that minority. The overwhelming majority can give the product 8s through 10s, but if a large enough mean-spirited minority gives it 1s, it will be pulled down. Put these two trends together and you have a system that has the potential (I don't know if it ever has) to take the power out of the hands of the average EN World voter and put it into the hands of outliers. All that said, I am certain that the goals in creating this system were worthy. Why? Because I've seen that the goals of every other part of the ENnies have been worthy. This is not meant as an indictment of anyone's motives. This is meant as a highly wonky discussion of voting schemes. The potential damage of this system, even if it were to be horribly abused, is of course limited. Experience shows that the nominees are all of the highest quality, so it's not like bad products are walking away with ENnies as is often the case with awards programs in other industries. However, I think the voting system could be altered to unambiguously put the power into the hands of the majority of voters.All of this is based on assumptions: 1) about voting patterns and 2) that you are not using some kind of weighting scheme already. If I am wrong, please hit me with the mighty rhetorical mallet of smackdown.</blockquote>

Verification