ENnies V - and beyond...

[Cthulhu's Librarian]

I don't see the need for external judges. These are the ENnies; why can't they come from the EN World community?

Exactly. The awards are sponsored by ENWorld, and the judges should come from ENWorld as well. We are the ones putting up the cash (or more precisely, Russ is putting up the cash) to put the awards together. We are the ones who do all the work. If anything, maybe 1 outside judge, but by no means will we be giving up a majority (3? That's never going to happen) to outside judges.

 

[Conaill]

The votes are normalised. Which I don't understand, but a statistician friend of mine who is responsible for it being that way assures me that it is a good thing.

Normalization could mean a number of things, but most likely it means subtracting the mean and dividing by the stanadrd deviation.

So someone who voted nothing but 10's and 1's would have excatly the same effect as someone who voted nothing but 10's and 9's, or someone who voted all 6's and 5's.

If so, I assume this is very different from what people *thought* was going on with the scores.

Note that you can still manipulate this system, by voting 10 for you favorite, 1 for your anti-favorite, and 5 or 6 for everything else. Note that if you want to manipulate the system, it will always be in your best interest to score *all* choices, even if you haven't even heard of them.

 

[Lela]

Ladies and gentlemen, how to cheat well at the ENies, by Conaill.

I assume switching to a simple ballot system has been considered and rejected? If you like the product, vote for it. If you don't, don't. Much like how we decide the judges. People could be allowed one vote per catagory or more.

 

[Crothian]

I assume switching to a simple ballot system has been considered and rejected? If you like the product, vote for it. If you don't, don't. Much like how we decide the judges. People could be allowed one vote per catagory or more.

We had that but people complained that since certain products like a Wizards book was just so better known then the other books, that it was unfair to the little guys.

 

[Conaill]

Ladies and gentlemen, how to cheat well at the ENies, by Conaill.

I'd prefer to call it "destructive testing" of the Ennies

There's a long history in fields such as cryptography of trying to find loopholes by which you can exploit a system, in order to come up with a system which is harder to exploit. The primary requirement for this approach to work is *openness*. Right now, I don't know if there's more than 2 or 3 people that know exactly how the Ennies are scored...

I assume switching to a simple ballot system has been considered and rejected? If you like the product, vote for it. If you don't, don't. Much like how we decide the judges. People could be allowed one vote per catagory or more.

Check out some of the discussion between AaronLoeb and myself on pages 4-6 of this thread. I think we both agree that letting people assign an actual *score* to the different choices (whether it's 1-10 or 1-5) is overkill, making it much more likely that some people will try to manipulate the voting. On the other hand, allowing only one vote per category is too restrictive as well.

Two reasonable compromises are to allow multiple votes per category (approval voting), or to *rank* the choices within a category (with different methods available to deal with unranked choices, and to derive a final winner based on those rankings). Both of these approaches are much less open to manipulation than the current one, and are less biased against "the little guy" than one vote per category....

 

[Ed Cha]

I don't see how it would.

Let us take for granted that under the current 1-10 voting range, some folks are (or may) bollix up things by voting "1" against a product. Our averages are thereby affected.

Now, change it to a 1-5 scale. This is pretty darned close to simply dividing the previous results by 2. If the voting pattern remains the same, the relative rankings remain the same, and there's no change from the 1-10 scale in effective results.

I am oversimplifying, leaving out normalizing the votes and the like. But the principle is the same - shrinking the range does not lessen the ability to vote against a product. The size of the range in a ranking system only changes how many decimal places out you have to go in order to determine the winner.

In order to eliminate the effect, you have to either change teh way people vote (eliminate voting against products in the population) or eliminate the ability to vote against a product at all.

Depending on the normalization scheme being used, the effect of all this can be minimized, but not eliminated entirely. And it may be a bugaboo - Have we actually confirmed that a significant number of folks are min/max voting? Do we have reason to suspect that they might do so in the future?

I'm not a mathematician, but I think it would mitigate suspect voting.

 

[Umbran]

I'm not a mathematician, but I think it would mitigate suspect voting.

My point being that the first thought is incorrect. In terms of human ego when looking at the final socres, the effects may be mitigated. But the effects one the winners and losers are not. Using a different range merely sets a breadth of scale. It does not change the mechanics, and thus doesn't change the final results.

Let us consider two products (A and B), and four voters (J, P, G, and R). R, for some reason, really wants B to win (or A to lose), no matter the actual merits of product A.

On a 1-10 scale, let's say the votes go like this:

     A  B
J    9  8
P    9  7
G    9  7
R    1  10
Avg  7  8

Here, we see B win. But, if R had voted more based upon actual merit, he might have given product A a 6. That would have given A a score of 8.25, and it would have won.

If we change this to a 1-5 scale, and map the scores, we get this:

     A  B
J    5  4
P    5  4
G    5  4
R    1  5
Avg  4  4.25

So again B wins. If R were voting like everyone else, he might have given A a 3, yielding a 4.5 average for A.

One is tempted to say that this is a contrived example, but that would be missing the point. The point here is merely to demonstrate the general principle that changing the range does not alter the end results. It does not matter what the votes are - if you do a basic mapping from one range to the other, the winners and losers will be the same people.

Sure, on paper getting a 4 out of 5, and losing to a 4.5 out of five, doesn't look so bad. Getting a 7 out of 10, and losing to an 8 may feel worse. But that's a trick of human perception. A loss is still a loss.

There are reasons to choose one range over another, but most of them are over in the human perception end of things. Mathematically, they're pretty much the same. Any scoring system like this will have a problem if some folks vote in a more analog mode (using the whole range), while others go for the digital min/max mode.

 

[Conaill]

My point being that the first thought is incorrect. In terms of human ego when looking at the final socres, the effects may be mitigated. But the effects one the winners and losers are not. Using a different range merely sets a breadth of scale. It does not change the mechanics, and thus doesn't change the final results.

Not entirely true. Narrowing the range of scores will help somewhat. At the extreme, you could narrow it down to a 1-2 score, which would be equivalent to approval voting (assuming you score all the choices). But you're right that narrowing the range from 1-10 to 1-5 doesn't really change the fundamental problem.

Scoring is a good idea if what you want as an end result is an actual *score* for all the candidates, like on the reviews page or on imdb.com. But if all you want is to designate a single *winner* (plus a second place in our case), allowing the voters to assign scores to all candidates grants the voters WAY too much flexibility - flexibility they can then use to manipulate the system!

There's always a tradeoff between in how much detail you allow the voter to express his opinion, and how easy it is to exploit the voting system. If everyone were to vote completely honestly, this would not be a problem, and we could allow them a 1-100 score for all I care. But unfortunately we can't make that assumption.

The classical one-person-one-vote gives the voter the absolute *minimum* amount of choice, which makes it hard to abuse, but also means the outcome may be a poor fit to what the voters actually want. A scoring method as is now used by the Ennies gives the voters the *maximum* amount of choice, meaning that if everone votes honestly the outcome should most closely match the voters desires, but making it way too easy to manipulate.

 

[Nisarg]

Hi,

I guess tehre are two fundamental issues here:

1. What is manipulating the vote? For example, I didn't just give high votes to the products I did like, I also gave low votes to products I didn't. White Wolf did not end up doing well on my scoresheet. Is that manipulation? Or is that what is intended? Because really, if the people running the ENnies didn't want that kind of thing to happen they really shouldn't use that scoring system. I was damn sure that there were others out there who were giving my favourite products low scores too, so why wouldn't I do the same, to balance things out?

2. Is it for some reason important for the Judges to know what product we hated as well as loved? Are we doing an equivalent of the razzies later? Because otherwise I don't see the point of a scoring system that ranks from 1-10, best to worst. It would seem more logical to vote simply for your first and second choices out of each list. That should be enough to allow a fair determination of a winner, and keep "manipulation" to a minimum.

Nisarg

 

[Umbran]

1. What is manipulating the vote?

Manipulating the vote is playing with the numbers of the voting system with the express purpose of trying to get your favored product to win (or your unfavored one to lose), without regard to the actual merits of the product.

If you did an honest assessment of the products, and came up with a low score, that's fine. If you hate one publisher, so you vote "1" on all it's products, and "10" on all the competitors, you're manipulating the vote.

The thing is, we don't know that anyone is manipulating the system. It is merely a possibility. It is a point that speaks to the credibility and security of the awards system, is all.

I was damn sure that there were others out there who were giving my favourite products low scores too, so why wouldn't I do the same, to balance things out?

The idea of the voting shouldn't be (IMHO) to "balance things out". It should be to give an award to the highest quality and most popular products. You're supposed to give your personal honest opinion, not what you think will be balancing out the opinions of others.

2. Is it for some reason important for the Judges to know what product we hated as well as loved?

Not at this time, no.