Upgrade your account to a Community Supporter account and remove most of the site ads.

Reply to thread

Message: <blockquote data-quote="EzekielRaiden" data-source="post: 9835757" data-attributes="member: 6790260">It turns out, analyzing a complex, multifaceted thing by simplifying it down to a number 1-10 isn't really the best survey design. They've been using this survey design, or others extremely similar to it, since 2012.If you don't trust the methods they're using right now, and think those methods are liable to produce more harm than good, what does that say? If the method hasn't changed in a decade and a half, and the only thing it does is sometimes cut out "the real duds", while sometimes leaving good work on the cutting room floor (e.g. playtest Sorcerer) or leaving in execrable crap (e.g. the 5.0 Berserker), what should we conclude?This is why I've said, for years and years, that the thing WotC needs is someone on staff, or at least a consultant, who is actually trained in statistics, and someone else who is on staff (or a consultant), who is actually trained in survey design and analysis.Because when you build a product based on statistical, mathematical rules, and "test" those rules by collecting survey data, I would say it is kind of important to have at least one person who actually knows how to do each of those things at a professional level.Oh, the duration doesn't help, to be sure. But you could collect data for a full year and still not get anything meaningful out of it if the survey itself is designed badly--whereas a well-designed survey, even if it is over a limited range, still has a good chance of giving you something useful, even if it's imperfect. No survey will ever be perfect, but they could be a damn sight better than they are.</blockquote>

Verification