What do we actually know about WotC's market research?

[wedgeski]

So, you are basically saying that if I answer a survey about what my partner or my children like, the information is as accurate as if they answer? Or the other way around?

If the 'thing' that you're talking about is a group activity like D&D, then I would say yes. I could for example tell you pretty well what my wife likes and doesn't like in a game of D&D. Similarly I could tell you what movies and music she likes, as we tend to both watch and/or listen to the same thing at the same time.

Any information that does not come from the primary source is second-hand by definition.

On this, nevertheless, I agree.

 

[nothing to see here]

Sorry to bore you here

So, you are basically saying that if I answer a survey about what my partner or my children like, the information is as accurate as if they answer? Or the other way around?

Any information that does not come from the primary source is second-hand by definition.


RC

Raven raises a good point -- many households have one member who answers the phone far more frequently than anyone else...fortunately there are ways around that. Most professional research companies ensure a degree of randomness within a household as well. For instance a common, easy to implement trick is to ask for the adult member of the household who's birthday falls closest to (random month) 1st, to participate in the survey as well.

And keep in mind that -- for most consumer purposes 'household's' aren't the units of measurement that people are interested in*. Household's don't buy game products, individual consumers do. As long as the person answering the phone is a randomly selected repreentative of game-buyers in general...your research is rocking and rolling.

On that point, it is also worth noting that the whole sample-size = confidence intreval notion does not apply in it's entirety here. The math behind statistics only works 'cleanly' assuming a completely random sample of a given population. Complete 'randomness' is next to impossible for any population, but for wide scale research (i.e. political voter intention research) you can get pretty darn close.

This is not the case here, however. Since the 1000 people sampled in WOTC's market research sample were nothing more than a (prbably random) selection of the 20,000 (or fewer)households that responded to the inital post card survey, then an important staistical rule regarding 'probability' of survey accureacy was broken -- namely that the population being sampled was not random, but self-selecting.

This does not invalidate the research, however. The leap of faith comes from your confidence that the 20,000 gamers contacted by the post cards are representative of everyone who plays D&D. Charle's Ryan's point is very accureate...using this or other fcustomer feedback mechanisms to create a sample-able sub population of the overall buying public is quite common as well as, usually, very accureate. Quite frankly, it is hard to to demand random sampling of the entire buying public in almost any industry (though the growth of the virtual marketplace will make that easier as time goes on). For one, you have to do some arcane statistics-fu and with regard to weighting your sample so that it corresponds, demographically, to the actual buying public.

That's probably more about modern research than you want to know. Sorry. It's not exactly shooting fish in a barrell, but neither is it all Smoke and Mirrors either.

*one exception remains Nielsen and other TV ratings systems where what is watched on TV is very much a communal 'household' choice.

 

[DM_Jeff]

I got an invite by email from a WotC marketing researcher to take part in a personal study of D&D players and how they play. It was for a specific date of GenCon 2000. D&D 3.0 had just come out, but they were seeming to prep for how they could support the game and WHO was actually playing these games and how.

I sat in a room with about 5 other individuals, male and female, ages 19 to 55. We answered as a group, tons of questions about our experience, play styles, who were DMs and players, what we liked, disliked, etc. And it wasn't just about rules, but the sociable aspect of gaming, table dynamics, visual tastes, our tendency to tweak stuff or play RAW, and this went on for 2 hours. The folks conducting the research (a woman and a couple of guys) seemed alert and interested but tired; they mentioned they were doing this all GenCon weekend with groups like ours all day long.

We remember it being a really positive experience. Frequently the researchers nodded to each other, as a "that what we expected" type thing, ad then occassionally we'd all answer in one direction which brought about lots of raised eyebrows that then spawned a bunch of new, tangent questions around that topic. Overall I was impressed at the methods used (they digitally recorded the whole meeting, tooks loads of notes, and had us fill out multiple question cards).

-DM Jeff

 

[Numion]

So, you are basically saying that if I answer a survey about what my partner or my children like, the information is as accurate as if they answer? Or the other way around?

Any information that does not come from the primary source is second-hand by definition.


RC

I was thinking more along the lines of non open ended questions, like how often does your group play, how large is your group, how long does your groups campaigns last. Granted, I don't know what they asked. Secondly I'm not sure how significant the effect would be even if what you said was true. I only know households with one player.

Still, what made you so suspicious of the research? I mean, it's not exactly rocket science, even though it's not simple either. I mean:

1. WotC does market research
2. WotC analyzes the data
3. WotC does new edition partly using that analysis
4. 3E is a resounding success
​

Where do you see the big gap in this? Obviously they got something right from the research.

Besides, wouldn't there be a better way to con the customer than a bogus market research, the existence of which (much less the results) isn't known to but, what, a 5% of D&D players?

 

[Joshua Randall]

I currently get about 4 e-mails a year inviting me to answer an online survey for WotC. (Can't remember how I signed up for this... was probably via a link on wizards.com a while back.)

The survey is usually the same in structure with only the specifics of the questions changing. For example, which of the following products do you own / have you heard of; rate the products you own on a numerical scale; why don't you own certain products (no interest, heard it was bad, don't play in FR / Eberron / Greyhawk / whatever). Also lots of question about minis. And, questions about how much you spent on D&D in the last month / 3 months / year.

 

[BlakeT]

They cast a summon geeks spell by opening up a can of soda while talking about Star Wars. Then they bind them by offering said geeks a soda and some chips. If that fails they bring out the pizza.

OK I'm not awake yet, but it would work!

 

[Gentlegamer]

IIRC, that kind of thing is pretty standard. If you can generate a representative sample, then you test with a far more management group size while still maintaining accuracy (of course, some margin for error is introduced).

I'm not really a statistics expert though.

A wise man once said that there are three kinds of lies: lies, damn lies, and statistics.

 

[Victim]

A wise man once said that there are three kinds of lies: lies, damn lies, and statistics.

79% of statistics is misleading.

 

[crazy_cat]

A wise man once said that there are three kinds of lies: lies, damn lies, and statistics.

79% of statistics is misleading.

But anything you read on the internet must be true.

 

[Numion]

A wise man once said that there are three kinds of lies: lies, damn lies, and statistics.

It's better to believe numbers than nothing at all.