ENnies - let's launch the voting booth!
- Thread starter Morrus
- Start date
Olgar Shiverstone
Legend
I, too , would be interested in hearing about the mechanics of tallying votes.
For example, are the votes weighted based on the number of people who respond? (If not, I want exactly one voter to vote 10 for my product.)
A people's votes normed before tallying? If I vote and my mean score is eight, and someone else has a mean score of five, with the same deviations, my votes are going to sway the scores upward, even though they really shouldn't -- I'm just biased toward higher scores.
This voting system actually results in an extremely complex multiple stakeholder decisionmaking problem. To be fair and accurate, it really requires the data to be analyzed in such a way as to determine a value curve for each voter, then reassess the scores based upon that value curve, then (assuming equal shareholder weights) decide on a method to weight vote volumes.
Given that it's too late to redesign the system, I'd recommend some of the following measures be used for tallying votes. These are a bit of a simplification, but it'll get us close to fair without calculating value curves or going into multi-attribute utility theory:
- Delete obvious outliers (someone who votes "1" on everything, then "10" on one product).
- Norm each individuals scores to an N(0,1) distribution before they're calculated. Individual values may not actually be normally distributed, so this is a simplification, but when we start adding votes the Central Limit Theorem will kick in and we'll end up wth a normal distribution anyway. This norms out individual biases.
- Sum normed votes. Calculate the required minimum votes to be a statistically representative sample of the population, and use that as a cut-off. A product with fewer votes than than is ineligible to win regardless of score. This ensures that enough votes are gathered to be a representative sample, while still allowing less familiar products a chance to complete.
- Compare the remaining products in each category via a one-factor analysis of variance, to ensure that the winner in each category is actually the winner (outside of the margin of variance). If so, you have a winner. If not, you'll have to either narrow the voter pool and reanalyze, or else declare a tie.
Have fun!
(Can you tell I'm working on a graduate engineering degree focused on applying decisionmaking processes to user evaluation of hardware/software?)
For example, are the votes weighted based on the number of people who respond? (If not, I want exactly one voter to vote 10 for my product.)
A people's votes normed before tallying? If I vote and my mean score is eight, and someone else has a mean score of five, with the same deviations, my votes are going to sway the scores upward, even though they really shouldn't -- I'm just biased toward higher scores.
This voting system actually results in an extremely complex multiple stakeholder decisionmaking problem. To be fair and accurate, it really requires the data to be analyzed in such a way as to determine a value curve for each voter, then reassess the scores based upon that value curve, then (assuming equal shareholder weights) decide on a method to weight vote volumes.
Given that it's too late to redesign the system, I'd recommend some of the following measures be used for tallying votes. These are a bit of a simplification, but it'll get us close to fair without calculating value curves or going into multi-attribute utility theory:
- Delete obvious outliers (someone who votes "1" on everything, then "10" on one product).
- Norm each individuals scores to an N(0,1) distribution before they're calculated. Individual values may not actually be normally distributed, so this is a simplification, but when we start adding votes the Central Limit Theorem will kick in and we'll end up wth a normal distribution anyway. This norms out individual biases.
- Sum normed votes. Calculate the required minimum votes to be a statistically representative sample of the population, and use that as a cut-off. A product with fewer votes than than is ineligible to win regardless of score. This ensures that enough votes are gathered to be a representative sample, while still allowing less familiar products a chance to complete.
- Compare the remaining products in each category via a one-factor analysis of variance, to ensure that the winner in each category is actually the winner (outside of the margin of variance). If so, you have a winner. If not, you'll have to either narrow the voter pool and reanalyze, or else declare a tie.
Have fun!
(Can you tell I'm working on a graduate engineering degree focused on applying decisionmaking processes to user evaluation of hardware/software?)
Last edited:
Cthulhu's Librarian
First Post
Very nice! That may be the easiest and most well designed online voting booth I have used. I had to vote for quite a few "Not familiar", but thats better than having to guess. I'm looking forward to the ceremony, so I can see how my tastes stand up against ENWorld as a whole. 
Voted!
Have to agree that it was a very good design, Florida could learn a thing or two from it.
Have to agree that it was a very good design, Florida could learn a thing or two from it.
KDLadage
Explorer
I will answer this... but not right now. Ask me again after this is over. I would not want to create a debate over how the current votes are tallied (should my personal ideas on how the voting should be handled differ from those that designed the system).Blacksway said:
Well, off the top of my head, you have:Blacksway said:
- Standard average (mean, median or mode) -- this results in things like a single vote of 10 being seen as a total vote of 10.
- Weighted average (mean, median or mode) -- with the number of votes a product gets getting into the mix as well.
- Slanted average (mean, median or mode) -- think of this as treating the vote as if it were the vote squared... such that 10's are 4-times the weight of a 5 and so on.
- Slanted/Weighted average (mean, median or mode) -- this is like combining slanted and weighted averages.
- Virtual Negatives -- take any of the voting schemes above. Now, treat a vote of 6 as a +1 vote; a vote of 5 becomes a -1 vote, a vote of 1 becomes a -5 vote; a vote of 10 becomes a +5 vote.
- Virtual Negatives + Weighted Average
- Virtual Negatives + Slanted Average
- Virtual Negatives + Weighted Average + Slanted Average
- and so on and so forth...
And this does not even get into the many ways, using the miriad of Statistical Tools this could go...
Olgar Shiverstone said:
Sorry. I barely recognise this post as English. Could you rephrase it in a way that a normal human being can understand it?
[Edit - Really. I just reread it. I don't think there's a single sentence in all that lot which makes any sense to me and I'm pretty literate!]
Last edited:
Fiery James
First Post
The booth seems to be working fine, although I'm unable to give a "20" rating to any FDP products.
What's that Take 20 rule again?
Best of luck, all.
I'm getting my bridesmaid dress all ready for the ceremony.
- James
What's that Take 20 rule again?
Best of luck, all.
I'm getting my bridesmaid dress all ready for the ceremony.
- James
der_kluge
Adventurer
Voted. It's amazing how little I know about the gaming products out there. I need to start spending more time at my FLGS.
On a side note, it seems to me that some vindictive folks could vote their favorite a '10', and then slap '1's into all the other candidates, thus skewing the results. Hopefully, people will be honest, but we'll see, I suppose.
On a side note, it seems to me that some vindictive folks could vote their favorite a '10', and then slap '1's into all the other candidates, thus skewing the results. Hopefully, people will be honest, but we'll see, I suppose.
Olgar Shiverstone
Legend
Sorry to use too much mathematical jargon. Let me see if I can translate some of this so that a person without a background in statistics can follow it.
This is just trying to define the basic theory that applies. A branch of decision making analysis called utility theory examines the consequences of people's choices when they are forced to make decisions based on a deterministic system -- like when assigning scores to a ranking of products. Although ranking systems are intended to be linear (each "step" in the score is worth the same value), people's individual value systems are not: the difference in quality between a "5" and "6" product that you vote for may not be the same as the difference between a "9" and a "10" product. This is what messes up strictly numerical rating systems, particularly at low numbers -- a "2" product is exactly twice as good as a "1", but a "3" is only 1.5x as good as a "2". Value theory (and it's extension, utility theory, which deals with uncertainty -- but really doesn't apply here, since there aren't any "maybe" answers that include uncertainty) is designed to correct desicion weighting -- the scores that people given -- based on their perceived value of the score. Each individual has a different value curve -- for one person, a "5" might be twice as good as a "4", while for another it might be 4x as good. Value theory enables all those scores to be compared equally -- without the correction, you're comparing apples to oranges, in essence, an it is possible for some people's votes to carry more weight than others.
But value theory is fairly complicated to apply, because it requires evaluating a set of tradeoffs for each person, so we can't apply it directly, here. What we can try to do is come close, with the goal that each person's vote essentially carries equal weight, and that products are ranked by their quality, not just popularity (or why else have the 0-10 rating?).
Central to this concept is the fact that individual rankings of a product don't directly assess the overall quality of the product -- you're actually estimating the quality of the product from a sampling of people that have used the product. Done correctly, you'll estimate the real quality of the product within a certain margin of error (essentially what polls do when they sample X number of people and report an answer +/- a certain amount).
First, get rid of anyone trying to screw up the voting system, by having a non-regular voting pattern (which Morrus is already doing). The "average" voter will have a certain voting distribution that can be described mathematically, within a certain margin of variability. Anyone who falls well outside that can be assumed to be trying to fix votes and should be deleted.
This concept is a little difficult to follow if you haven't had stats, but essentially when two people rate a product, their rating's aren't equal. Even if you have a 10-point rating scale, no two people are going to use the entire scale in the same way (because of the value information I presented above). Some are biased toward high scores, some are biased toward low scores, some might have a tight grouping (only score 4-6, for example), others might use the whole range. Differing variance and mean (voting bias) can skew results, and cause certain poeple's votes to effectively carry more weight. With a large enough sample of votes, this tends to be reduced somwhat -- but why not correct it right off the bat?
We can "correct" everyone's votes so that everyone uses the same distribution -- a normal (Gaussin, bell) curve with a mean (average) score of 0 and a variance of 1 (ie, N(0,1)). If you take all the ratings a person gives, calculate the mean and standard deviation of those scores , you can arrive at a corrected score for each product by taking the individual's score, subtracting the mean score, and dividing by the standard deviation. This generates a set of scores that range from -4 to +4, distributed along a bell curve -- and if done for every individual, their scores will be distributed along the same curve. A -4 correleates to the intended lowest score, +4 to the highest, and the middle value -- 0 -- will now correlate to the intended "average" score: 5.
That way, when you add their scores, you've eliminated individual bias, to ensure that everyone's score means the same thing.
A sample -- set of votes -- has to be big enough to generate a truly representative sampling. If a product has sold 1 million copies, for example, and you only get 10 ratings, are those ten truly indicative of the quality of the product? Or did only the biggest whiners/fanboys vote?
Morrus has established a cutoff, which is good. You can calculate an exact number needed, based on how accurate you want to be -- but for our purposes a swag estimate will probably work.
We want to make sure the winner is really the winner, without question. Say, for example, you have Product #2 that gets 10 votes, all 5's (we'll ingore norming for the moment, and assume these are normed scores). The total score is 50, mean 5. Product #2 gets 4 10's, 4 2's, and 2 1's: total 50, mean 5. Product #3 gets 5 7's, a 6, 2 2's, and 2 1's: total 47, mean 4.7. Who wins? Strictly by total or mean score, 1 and 2 tie, both slightly better than 3 -- but is that how we should judge it? Product 3 has more "above average" scores than either of the other two products, for example. Because of variance, the apparent winner may not be the actual winner.
As sample sizes get very large, it's possible to construct scenarios where widely varying scores are actually the same due to variance. That's the purpose of the ANOVA, to test that the winning score is actually statistically different than the others. There's more to it than that, of course -- I'm trying to avoid any deeper discussion. The point is -- make sure everyone's vote counts equally, and that the winner is really far enough ahead to be the winner.
There's quite an involved science behind ratings and evaluations. Multiple voter (also know as stakeholder) systems which involve individual rating schemes are one of the most complicated systems to get to work in a truly fair manner -- be glad elections are usually held on a "one-man, one-vote" plurality/majority system.
If you're interested in more reading about decision making and value theory, there's a great little book written purely in layman's terms, called Smart Choices, by Hammond, Keeney, and Raiffa.
Hope I haven't bored everyone to tears. Thanks for bearing with my pedantry.
This is just trying to define the basic theory that applies. A branch of decision making analysis called utility theory examines the consequences of people's choices when they are forced to make decisions based on a deterministic system -- like when assigning scores to a ranking of products. Although ranking systems are intended to be linear (each "step" in the score is worth the same value), people's individual value systems are not: the difference in quality between a "5" and "6" product that you vote for may not be the same as the difference between a "9" and a "10" product. This is what messes up strictly numerical rating systems, particularly at low numbers -- a "2" product is exactly twice as good as a "1", but a "3" is only 1.5x as good as a "2". Value theory (and it's extension, utility theory, which deals with uncertainty -- but really doesn't apply here, since there aren't any "maybe" answers that include uncertainty) is designed to correct desicion weighting -- the scores that people given -- based on their perceived value of the score. Each individual has a different value curve -- for one person, a "5" might be twice as good as a "4", while for another it might be 4x as good. Value theory enables all those scores to be compared equally -- without the correction, you're comparing apples to oranges, in essence, an it is possible for some people's votes to carry more weight than others.
But value theory is fairly complicated to apply, because it requires evaluating a set of tradeoffs for each person, so we can't apply it directly, here. What we can try to do is come close, with the goal that each person's vote essentially carries equal weight, and that products are ranked by their quality, not just popularity (or why else have the 0-10 rating?).
Central to this concept is the fact that individual rankings of a product don't directly assess the overall quality of the product -- you're actually estimating the quality of the product from a sampling of people that have used the product. Done correctly, you'll estimate the real quality of the product within a certain margin of error (essentially what polls do when they sample X number of people and report an answer +/- a certain amount).
First, get rid of anyone trying to screw up the voting system, by having a non-regular voting pattern (which Morrus is already doing). The "average" voter will have a certain voting distribution that can be described mathematically, within a certain margin of variability. Anyone who falls well outside that can be assumed to be trying to fix votes and should be deleted.
This concept is a little difficult to follow if you haven't had stats, but essentially when two people rate a product, their rating's aren't equal. Even if you have a 10-point rating scale, no two people are going to use the entire scale in the same way (because of the value information I presented above). Some are biased toward high scores, some are biased toward low scores, some might have a tight grouping (only score 4-6, for example), others might use the whole range. Differing variance and mean (voting bias) can skew results, and cause certain poeple's votes to effectively carry more weight. With a large enough sample of votes, this tends to be reduced somwhat -- but why not correct it right off the bat?
We can "correct" everyone's votes so that everyone uses the same distribution -- a normal (Gaussin, bell) curve with a mean (average) score of 0 and a variance of 1 (ie, N(0,1)). If you take all the ratings a person gives, calculate the mean and standard deviation of those scores , you can arrive at a corrected score for each product by taking the individual's score, subtracting the mean score, and dividing by the standard deviation. This generates a set of scores that range from -4 to +4, distributed along a bell curve -- and if done for every individual, their scores will be distributed along the same curve. A -4 correleates to the intended lowest score, +4 to the highest, and the middle value -- 0 -- will now correlate to the intended "average" score: 5.
That way, when you add their scores, you've eliminated individual bias, to ensure that everyone's score means the same thing.
A sample -- set of votes -- has to be big enough to generate a truly representative sampling. If a product has sold 1 million copies, for example, and you only get 10 ratings, are those ten truly indicative of the quality of the product? Or did only the biggest whiners/fanboys vote?
Morrus has established a cutoff, which is good. You can calculate an exact number needed, based on how accurate you want to be -- but for our purposes a swag estimate will probably work.
We want to make sure the winner is really the winner, without question. Say, for example, you have Product #2 that gets 10 votes, all 5's (we'll ingore norming for the moment, and assume these are normed scores). The total score is 50, mean 5. Product #2 gets 4 10's, 4 2's, and 2 1's: total 50, mean 5. Product #3 gets 5 7's, a 6, 2 2's, and 2 1's: total 47, mean 4.7. Who wins? Strictly by total or mean score, 1 and 2 tie, both slightly better than 3 -- but is that how we should judge it? Product 3 has more "above average" scores than either of the other two products, for example. Because of variance, the apparent winner may not be the actual winner.
As sample sizes get very large, it's possible to construct scenarios where widely varying scores are actually the same due to variance. That's the purpose of the ANOVA, to test that the winning score is actually statistically different than the others. There's more to it than that, of course -- I'm trying to avoid any deeper discussion. The point is -- make sure everyone's vote counts equally, and that the winner is really far enough ahead to be the winner.
There's quite an involved science behind ratings and evaluations. Multiple voter (also know as stakeholder) systems which involve individual rating schemes are one of the most complicated systems to get to work in a truly fair manner -- be glad elections are usually held on a "one-man, one-vote" plurality/majority system.
If you're interested in more reading about decision making and value theory, there's a great little book written purely in layman's terms, called Smart Choices, by Hammond, Keeney, and Raiffa.
Hope I haven't bored everyone to tears. Thanks for bearing with my pedantry.
Last edited:
Similar Threads
