Reply to thread

Message

<blockquote data-quote="Conaill" data-source="post: 1762993" data-attributes="member: 1264">Note that some other entries in the top 20 have *many* more 5-star votes than the 6 for Slaine and Midnight. For example, Mutants &amp; Masterminds has a 11 5-star votes and only one single 4-star. It's not that M&amp;M has been &quot;torpedoed&quot; as Psion so colorfully puts it, it's more that some of the top products have only had a very small number of votes, so their rating isn't quite as reliable as for the products with many more votes.See, this is a good example why more &quot;professional&quot; rating services like <a href="http://www.imdb.com/chart/top" target="_blank">imdb.com</a> use something called a &quot;Bayesian estimate&quot; to compensate for differences in the number of votes for each candidate. I won't go into details of why the formula has the shape it does, but the upshot is that it's equivalent to adding a small number of &quot;pseudo&quot;-votes with a score equal to the overall average score.For example, the average score for print reviews is around 3.85 , so let's add 5 &quot;3.85 star&quot; votes to each product. Sum of scores for Slaine would be 6 x 5.00 + 5 x 3.85 = 49.23, for an adjusted average score of 4.48. Sum of scores for M&amp;M would be 11 x 5.00 + 1 x 4.00 + 5 x 3.85 = 78.23, for an adjusted average score of 4.60.So if you take into account the fact that M&amp;M has twice as many votes as Slaine, and only one single 4-star vote, M&amp;M should be ranked higher than M&amp;M.Here's what the top 20 would look like, using a Bayesian estimate a la imdb.com:[code]1&nbsp; &nbsp; Mutants &amp; Masterminds&nbsp; &nbsp; Green Ronin Publishing&nbsp; &nbsp; 4.602&nbsp; &nbsp; Tome of Horrors&nbsp; &nbsp; &nbsp; &nbsp;&nbsp; Necromancer Games&nbsp; &nbsp; 4.543&nbsp; &nbsp; Manual of the Planes&nbsp; &nbsp; Wizards of the Coast&nbsp; &nbsp; 4.544&nbsp; &nbsp; 2000AD: Sla'ine RPG&nbsp; &nbsp; Mongoose Publishing&nbsp; &nbsp; 4.485&nbsp; &nbsp; Midnight&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Fantasy Flight Games&nbsp; &nbsp; 4.486&nbsp; &nbsp; Crooks!&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; Green Ronin Publishing&nbsp; &nbsp; 4.427&nbsp; &nbsp; Lords o/tNight:Vampires&nbsp; &nbsp; Bottled Imp Games&nbsp; &nbsp; 4.428&nbsp; &nbsp; Starfarers Handbook&nbsp; &nbsp; Fantasy Flight Games&nbsp; &nbsp; 4.429&nbsp; &nbsp; Book of the Righteous&nbsp; &nbsp; Green Ronin Publishing&nbsp; &nbsp; 4.4210&nbsp; &nbsp; The Book of Taverns&nbsp; &nbsp; Necromancer Games&nbsp; &nbsp; 4.4011&nbsp; &nbsp; Call of Cthulhu d20&nbsp; &nbsp; Wizards of the Coast&nbsp; &nbsp; 4.4012&nbsp; &nbsp; Scarred Lands: Ghelspad&nbsp; &nbsp; Sword &amp; Sorcery Studios&nbsp; &nbsp; 4.3713&nbsp; &nbsp; Lords o/t Night: Liches&nbsp; &nbsp; Bottled Imp Games&nbsp; &nbsp; 4.3714&nbsp; &nbsp; The Tomb of Abysthor&nbsp; &nbsp; Necromancer Games&nbsp; &nbsp; 4.3515&nbsp; &nbsp; Monsternomicon Vol. I&nbsp; &nbsp; Privateer Press&nbsp; &nbsp; &nbsp;&nbsp; 4.3516&nbsp; &nbsp; Forgotten Realms CS&nbsp; &nbsp; Wizards of the Coast&nbsp; &nbsp; 4.3317&nbsp; &nbsp; Legions of Hell&nbsp; &nbsp; &nbsp; &nbsp; Green Ronin Publishing&nbsp; &nbsp; 4.3318&nbsp; &nbsp; Hammer &amp; Helm&nbsp; &nbsp; &nbsp;&nbsp; Green Ronin Publishing&nbsp; &nbsp; 4.3219&nbsp; &nbsp; Plot and Poison&nbsp; &nbsp; &nbsp; &nbsp; Green Ronin Publishing&nbsp; &nbsp; 4.3220&nbsp; &nbsp; Unearthed Arcana&nbsp; &nbsp; &nbsp; &nbsp;&nbsp; Wizards of the Coast&nbsp; &nbsp; 4.2921&nbsp; &nbsp; Judge Dredd RPG&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Mongoose Publishing&nbsp; &nbsp; 4.29[/code]Note how all the 5- and 6-vote entries moved down, whereas entries with many more 5-star reviews (such as MotP, Starfarer's HB and BotR) moved up. FRCS, with a whopping 17 reviews of which 11 5-star, moved up 13 places, from #29 to #16, whereas Hammer &amp; Helm and Plot &amp; Poison dropped 11 places because they only barely squeaked by the 5-vote minimum.So what do you all think... does this ranking make more sense to you? From a statistical point of view, it should be more reliable...</blockquote>

[QUOTE="Conaill, post: 1762993, member: 1264"] Note that some other entries in the top 20 have *many* more 5-star votes than the 6 for Slaine and Midnight. For example, Mutants & Masterminds has a 11 5-star votes and only one single 4-star. It's not that M&M has been "torpedoed" as Psion so colorfully puts it, it's more that some of the top products have only had a very small number of votes, so their rating isn't quite as reliable as for the products with many more votes. See, this is a good example why more "professional" rating services like [url=http://www.imdb.com/chart/top]imdb.com[/url] use something called a "Bayesian estimate" to compensate for differences in the number of votes for each candidate. I won't go into details of why the formula has the shape it does, but the upshot is that it's equivalent to adding a small number of "pseudo"-votes with a score equal to the overall average score. For example, the average score for print reviews is around 3.85 , so let's add 5 "3.85 star" votes to each product. Sum of scores for Slaine would be 6 x 5.00 + 5 x 3.85 = 49.23, for an adjusted average score of 4.48. Sum of scores for M&M would be 11 x 5.00 + 1 x 4.00 + 5 x 3.85 = 78.23, for an adjusted average score of 4.60. So if you take into account the fact that M&M has twice as many votes as Slaine, and only one single 4-star vote, M&M should be ranked higher than M&M. Here's what the top 20 would look like, using a Bayesian estimate [i]a la[/i] imdb.com: [code] 1 Mutants & Masterminds Green Ronin Publishing 4.60 2 Tome of Horrors Necromancer Games 4.54 3 Manual of the Planes Wizards of the Coast 4.54 4 2000AD: Sla'ine RPG Mongoose Publishing 4.48 5 Midnight Fantasy Flight Games 4.48 6 Crooks! Green Ronin Publishing 4.42 7 Lords o/tNight:Vampires Bottled Imp Games 4.42 8 Starfarers Handbook Fantasy Flight Games 4.42 9 Book of the Righteous Green Ronin Publishing 4.42 10 The Book of Taverns Necromancer Games 4.40 11 Call of Cthulhu d20 Wizards of the Coast 4.40 12 Scarred Lands: Ghelspad Sword & Sorcery Studios 4.37 13 Lords o/t Night: Liches Bottled Imp Games 4.37 14 The Tomb of Abysthor Necromancer Games 4.35 15 Monsternomicon Vol. I Privateer Press 4.35 16 Forgotten Realms CS Wizards of the Coast 4.33 17 Legions of Hell Green Ronin Publishing 4.33 18 Hammer & Helm Green Ronin Publishing 4.32 19 Plot and Poison Green Ronin Publishing 4.32 20 Unearthed Arcana Wizards of the Coast 4.29 21 Judge Dredd RPG Mongoose Publishing 4.29 [/code] Note how all the 5- and 6-vote entries moved down, whereas entries with many more 5-star reviews (such as MotP, Starfarer's HB and BotR) moved up. FRCS, with a whopping 17 reviews of which 11 5-star, moved up 13 places, from #29 to #16, whereas Hammer & Helm and Plot & Poison dropped 11 places because they only barely squeaked by the 5-vote minimum. So what do you all think... does this ranking make more sense to you? From a statistical point of view, it [i]should[/i] be more reliable... [/QUOTE]

Verification