An "open letter" to WotC staff on survey design

[jeffh]

I would add to your list, questions that carry false presuppositions. For example, the one in the latest Legends & Lore that assumes that major changes in tone must be either (a) linked to levelling, or (b) nonexistent, ignoring the possibility many of the comments on that poll mention that the two have bugger-all to do with one another.

 

[Dice4Hire]

In any targeted survey there are always bias risks but I don't think they are likely to be too bad in this case as there are not likely to be large sections of the D&D market segment who do not have internet access and are unaware of the 5e ongoing design process. While there is a risk is deliberate mischief in polls like this there are ways to detect and filter out that bad data. Not that they seem to be employing them...

They are missing a huge potential market segment, the people who do not yet play D&D.

 

[Mishihari Lord]

The biggest problem is the sampling technique. The polls are answered by people who come to the WOTC website who like participating in polls. This is such a severe flaw that the rest of the OP's points are irrelevant. There is no way the results will have any statistical significance.

Since any competent marketer knows enough about polls to not make such a mistake, I have to think that the polls have another purpose entirely. My guess is that they're there to set a direction to the responses to the article that the writers think will yield useful information.

Edit: Whoops, looks like a lot of people already said this. Oh well.

 

[kevtar]

The biggest problem is the sampling technique. The polls are answered by people who come to the WOTC website who like participating in polls. This is such a severe flaw that the rest of the OP's points are irrelevant. There is no way the results will have any statistical significance.

Since any competent marketer knows enough about polls to not make such a mistake, I have to think that the polls have another purpose entirely. My guess is that they're there to set a direction to the responses to the article that the writers think will yield useful information.

Edit: Whoops, looks like a lot of people already said this. Oh well.

There are a variety of sampling methods: random sampling, purposeful sampling, convenience sampling - and they each have their merit depending on the situation. In this case, I would argue the authors are using convenience sampling, and that approach can still yield useful data.

However, you make an interesting point in that the surveys may have less to do with feedback and more to do with responses to the article. I do wonder though, especially with the recent L&L column by Mike, to what degree the informal surveying being conducted has on actual design decisions. For example, Mike mentioned that his question on Twitter was the impetus for his question regarding when players were "done" with their characters. The question now is, is this just a mental exercise or do the responses to these questions actually affect design work?

 

[GX.Sigma]

You should email this to Wizards of the Coast if you haven't already.

 

[Andor]

They are missing a huge potential market segment, the people who do not yet play D&D.

While that's true, those people are not capable of giving informed answers about favored playstyle questions, so what would be the point in targeting them at this stage?

You handle them later, during the iterative design process by flagging groups who list new-to-d&d members for special attention to responses by those members, and by direct testing (probably in-house) of new gamers reacting to the product during the style and polish phase.

Really though, attracting newcomers is more in the hands of marketing than the game designers. They are already gunning for a simplified system. As long as the box is attractive, the barrier to entry isn't too high, and the language is not obfuscatory the designers do not have much need to target "virgins" specifically. The days when D&D was an unknown and socially fringy game are gone. If they can fire up the base, we existing players will recruit. At army bases, colleges, clubs. You know, like a cult.

 

[BobROE]

I think you (and others) are assuming that these little surveys are being used for serious market analysis and research. While I could be wrong, it seems to me that they are more of an effort to generate reader participation, involvement, and debate. It may be that the results are merely glanced at, or used mainly to show readers past results. I do believe WOTC has done serious market research in the past, and knows, largely, how to conduct it.

But someone is devoting time to writing these polls, coming up with questions, coming up with answers, etc.

Why not use that time to write up polls that could generate meaningful data?

 

[howandwhy99]

Uh, this thread's a bit off putting altogether. But I agree Wizards could be served by enlisting more professional market research in their plans.

The decision to playtest has been widely accepted and public relations have improved enormously, so I wouldn't change those.

 

[kevtar]

Uh, this thread's a bit off putting altogether. But I agree Wizards could be served by enlisting more professional market research in their plans.

The decision to playtest has been widely accepted and public relations have improved enormously, so I wouldn't change those.

I agree public relations have improved enormously, and I hope that continues. In fact, if they can improve their survey design that will only enhance current public relations. As far as off-putting, I guess that is up to individual interpretation, but it is a critique (and I hope a constructive one at that).

 

[UngeheuerLich]

These are good tips for someone seriously wanting to analyze those polls. And in later polls they should clearly pay attention.
Right now I am sure, those polls are just added after writing the article by the writer. Clearly intended to just add some communication instrument to the community reading the article.

I really doubt, that those polls are even analyzed at all.

I hoope though, that later there will be some serious polls. Put together by a professional. Independant from an article that precedes each question. (Which inherently makes those polls biased!)