The WotC Polls

[Stormonu]

It seems like every time I read one of the articles on the WotC site about 5E, I cringe to see what the polls are going to be. From others comments, I don't feel like I am alone in how ill-thought out these polls really seem.

It's not that I want them to stop the polls, it's just that the options they seem to put in are really slanted towards their own opinions - not so much that they are attempting to find out what folks want, so much as to find out who agrees with them.

I really wish that when they put the articles together, that there was more thought and care put into both the wording and the options in the polls. Surely, they want meaningful data - I just worry that's not what they're really getting.

 

[BobTheNob]

To a degree, I agree with you. Too often the respon ses simply dont match what I want to say.

 

[Stormonu]

I want to point to Kevtar's thread about poll design here - I really think those writing up the articles consider his statements. I don't think the polls are attempting to be scienctific, but I do think they could be better.

http://www.enworld.org/forum/new-ho...letter-wizards-coast-staff-survey-design.html

 

[mkill]

Cut WotC some slack. They're showing effort. Yes, the polls could be better, but it's the first time a game company does that on such a large scale. I think they are on the right track, and I think they'll improve when they know what works and what doesn't.

It's time we stop acting like the Unpleasable Fanbase that we are.

 

[Andor]

Cut WotC some slack. They're showing effort. Yes, the polls could be better, but it's the first time a game company does that on such a large scale. I think they are on the right track, and I think they'll improve when they know what works and what doesn't.

It's time we stop acting like the Unpleasable Fanbase that we are.

No. No this isn't the first large scale research a gaming company has done. It's not even the first large scale research WotC has done. Not by a long shot.

Polling is a science not an art, although it does take a deft touch. There are any number of companies out there that do this for a living, and they are delighted to provide consultants to both shape polls and analyze the data which is not nearly as simple as you might think.

And I invite you to consider that the feedback mechanism for "Finding out what works and what doesn't" is that 5e succeeds or fails. This not not a speedy, precise or low-risk feedback mechanism.

Perhaps, given the number of professional scientists and market researchers comment on the excreable nature of these polls, you might consider that the people who do this for a living should perhaps have a hand in it, allowing the game designers more time to do what they do for a living. These are not armchair game designers whining that Paladin pokemounts are full of win and the system should be based on the d13. These are professionals providing free commentary and feedback in their own area of expertise, which is clearly not an area of expertise on the part of Mike, Monte, or you.

 

[Ratskinner]

I find the polls highly variable. Some just seem to be implemented poorly, as in "This question isn't a good subject for a poll" or "These options don't represent the full range of possible responses" or "Options 1 is the only option a sane person could select." Others seem rather tightly focused. Especially if some of the blog posts are taken into account, they range from truly horrible to fair/good.

However, I don't agree with all the harsh criticism of these polls as a group which are leveled by some of the other posters. I sometimes work on stats related to social science research issues, and trust me there's plenty of policy decisions affecting more important areas of life than rpgs that are based off of a lot worse than what we've seen from WOTC. It seems to me that a decent amount of that criticism is conflated with negative feelings or fears spawned from strained inferences from the article text.

Just my uninformed opinion, though. Its hard to judge such brief polls when you don't know the other end of what they are trying to find out and what they already know.

 

[Ratskinner]

And I invite you to consider that the feedback mechanism for "Finding out what works and what doesn't" is that 5e succeeds or fails. This not not a speedy, precise or low-risk feedback mechanism.

That's actually where I disagree. I really doubt that that's what's on the line. I more suspect that many of these polls are being used to get relatively sloppy input on where they should focus current efforts for the next round of playtests...and not much else. They are very early in the process for this edition and we haven't even had main thrust of their open playtesting effort yet. In addition to the core/basic rules, they also need to consider which modules will be most important to playtest the most/early. If that's the case, some of the more biased/lame polls may have just been to find out if they should put effort into Module X.

 

[Andor]

That's actually where I disagree. I really doubt that that's what's on the line. I more suspect that many of these polls are being used to get relatively sloppy input on where they should focus current efforts for the next round of playtests...and not much else. They are very early in the process for this edition and we haven't even had main thrust of their open playtesting effort yet. In addition to the core/basic rules, they also need to consider which modules will be most important to playtest the most/early. If that's the case, some of the more biased/lame polls may have just been to find out if they should put effort into Module X.

Admittedly there was a bit of hyperbole there, but there really isn't any other direct feedback mechanism for them to evaluate the effectiveness/accuracy of their polls.

Plus the fact that they are clearly not employing good market research tools at this stage makes me worried that they won't have the proper mechanisms in place to judge the feedback they do get from the play testers, and that could well be disastrous. There is, after all, no point in doing a proper iterative design process if your test-analyze-revise cycle is bollixed by lousy analysis.

 

[Fanaelialae]

The polls seem like they have room for improvement, but I also think the designers have received excessive flack regarding them. They are game designers, not market analysts. IMO, it's unreasonable to expect them to design perfect polls. And really, they're the only ones who know how useful or not the polls are to them; the best we can do is make assumptions based on extrapolation. It seems like some people would rather they didn't do any polls at all, which strikes me as rather counterproductive thinking.

 

[Dice4Hire]

No. No this isn't the first large scale research a gaming company has done. It's not even the first large scale research WotC has done. Not by a long shot.

Polling is a science not an art, although it does take a deft touch. There are any number of companies out there that do this for a living, and they are delighted to provide consultants to both shape polls and analyze the data which is not nearly as simple as you might think.

And I invite you to consider that the feedback mechanism for "Finding out what works and what doesn't" is that 5e succeeds or fails. This not not a speedy, precise or low-risk feedback mechanism.

Perhaps, given the number of professional scientists and market researchers comment on the excreable nature of these polls, you might consider that the people who do this for a living should perhaps have a hand in it, allowing the game designers more time to do what they do for a living. These are not armchair game designers whining that Paladin pokemounts are full of win and the system should be based on the d13. These are professionals providing free commentary and feedback in their own area of expertise, which is clearly not an area of expertise on the part of Mike, Monte, or you.

I think you are trying to hold WOTC to too high a standard. These polls are informal, and are part of a written article. WOTC also does more professional level polls periodically.

And how much time would these professional poll designers take to do professional polls, and how much would it cost? Both would be prohibitive, I am sure.