ENnies voting methodology

[anonystu]

Monte posted in a rave this morning about the ENnies:

People point out that the drawback is that you can vote 10 for your favorite product and 1 for all the rest, skewing the vote. I don't think that will actually be too much of a problem -- and it's something the moderators can watch for, in any case. The potential problem that I see is that the system favors products with fewer votes. For instance, a product that gets 99 votes of 7 and one vote of 8 receives an average rating of 7.01, which beats out a product that gets 5,000 votes of 7, generating an average of a straight 7. (Again, this assumes my understanding of the process is correct.) But that's only a potential scenario.

I posted in return on that message board:

Monte makes a very good point that the system may not account for different numbers of votes, but this is easily rectified by using a bayesian estimate, the system that you've most likely run into as the system that the IMDB uses to determine what the top-user-rated movies of all time are:

Check out this, for a not too technical explanation:

www.wowwebdesigns.com/formula.php

The technique is quite simple to explain: entries with low amount of votes are pulled towards the mean, because you should be less confident that those are representative of true preference, instead of just random sampling.

It's a quite easy solution to that issue: I'll email Morrus about it, but hey, you can talk to him too! I don't think this is a change that should happen this year: switching voting schemes mid-vote is as shady as it comes, but it's a good thing going forward.


So are the ENnies using a bayesian estimate formula to determine the winners? What do people think of switching to this method if it's not being used?

 

[Hand of Evil]

I am sure there is more to the ENnies than the vote.

 

[anonystu]

You mean: "more to the ENnies voting methodology than a simple average"? Any guesses what it is? I could imagine, actually, a bunch of interesting mechanisms based around comparisons, but I think simple average is probably the most likely mechanism.

 

[Piratecat]

Stu, I know that Morrus is working with a statistician in order to make sure that things work the way they're supposed to. If you want to offer your advice, I'm sure it would be useful!

- Kevin

 

[anonystu]

Cool! I should have had more faith that Morrus would be a wise fellow about this. I've forwarded him the link to the Bayesian estimate thing already. If I think of anything else, I'll be sure to pass it along.

Maybe also some collective enworld thought could help, so I'll post my initial thoughts here:

There are two main issues that I can think of, but no good answers off hand:

1) Do you attempt to detect or remove people who try to spike votes?

I'm not talking at this point about people who attempt to get around IP limitaitons, or do other sorts of blatant attempts at cheating, because those should be stopped pretty clearly, but rather, if a vote comes across where every category has one product at 10, and four products at 1, do you accept these as a casualty of the voting system, or do you try to mediate the person's attempt to make their vote count "more" somehow? Note that the solution to this may exist by answering the next question.

2) Can you do anything other than averages?

Consider these three scenarios:
Product A is the only product in a category to get a vote: it gets a 6.

Product A gets a 6, and Product B gets a 3, no other products get votes.

Product A gets a 6, Product B gets a 7, Product C gets a 8, Product D gets 9, and Product E gets a 10.

Is each 6 the same six? It's easy to see how you'd like to view those sixes in different lights, but it's tricky. It is my opinion, the greatest strength of this voting system (that it encourages you to only report on products you know), that makes it hardest here: there are lots of viable systems I could think of if everybody was required to vote on all products.

However, the benefit in honesty (in that I can just report on what I've seen), is so much bigger than any silly rank-order.


c) Can Morrus ethically change the voting tabulaiton system mid vote, or after vote?

My initial statement was slightly overbroad: there are a few ways where you can ethically do this, in my humble opinion.

1) If Morrus hasn't had any exposure to the votes, it's fine.
2) If Morrus has only had very limited and peculiar exposure to the votes, it's still fine (For example, if he only glances by IP addresses to check to make sure that people aren't spamming).
3) If Morrus looks at the votes without knowing what votes go where, it's fine.
4) If Morrus looks at small sample sizes, it's still okay, because it's hard to know how the race will go from a small sample size (although small is small: more than 30 or so votes looked at it in a category will probably give you a good idea of a distribution).

Obviously, even if he's not following these guidelines, Morrus is a good guy: I trust him not to monkey with the parameters to make his secret organization take over the world (or if he does, at the least, offer me a cool glass of water first). These are more general guidelines, on how altering rankings should go.

Hmm. That's a start for any armchair statisticians like myself.

 

[Morrus]

OK, I warn you in advance that I know less about statistics than your average cucumber does, but I do trust those those advice I've received and those who have implemented the system.

Some of the mathematical jargon associated with the system includes "normalisation", "standard deviation" etc. You may well know what those mean more than I do.

Also included are cut-off limits (i.e. minimum number of votes to qualify, which means that a product with 10 votes but all high won't beat a product with thousands of votes but with a lower rating) and manual rejection of what I consider to be "suspicious voting patterns".

The last thing, the "suspicious voting patterns", is the one which I imagine would cause concern if any any of it does, so I'll explain. Given a sample of 200 thousand million billion cajillion voters, I *might possibly* believe that there was one person there who genuinely thought that all but one of the nominated products were truly awful and worth only 1 point. That's "might" and "possibly" - and even then I'd raise an eyebrow.

Given that the number of voters is siginificantly less than 200 thousand million billion cajillion, I am rejecting votes such as 10/1/1/1/1 (and others, but listing them would defeat the point of doing it). I won't know what the votes are for or who made the votes, but I will be able to see the pattern. On this issue, I need to rely on everyone for a certain level of trust - similar to the trust I have in those same people to vote honestly.

Having said that, it really isn't an issue. I think we have a pretty honest set of votes.

 

[kingpaul]

Just out of curiosity, as the voting booths are now closed. how did the "unsure" selection affect the voting, if at all? And when will folks know if their product was even included in the final listing for votes? (I ask this one mainly for the NBoNPCs, as it may not be popularly known, and might not meet the requisite votes).

 

[Morrus]

Just out of curiosity, as the voting booths are now closed. how did the "unsure" selection affect the voting, if at all? And when will folks know if their product was even included in the final listing for votes? (I ask this one mainly for the NBoNPCs, as it may not be popularly known, and might not meet the requisite votes).

None of them failed to meet the minimum vote requirement.

The "Not familiar" option was used a lot, I'm glad to say.

 

[kingpaul]

None of them failed to meet the minimum vote requirement.

The "Not familiar" option was used a lot, I'm glad to say.

Whoo hoo! See you Friday Morrus!