What do we actually know about WotC's market research?

[werk]

So, my question is this: What do we actually know about WotC's market research?

I know that I signed up to take surveys from marketing, but by the time I get the survey invite email and follow the link, the survey is closed.

 

[Lanefan]

1)A sample of 1000 is a big, reasonable sample. Many, many of the studies I've done over the years have been with smaller samples -- honestly, a sample size of about 250 to 300 is usually enough for a high level of confidence in your results, although on a big "strategic" study, like the landmark study that fed into the development of 3E, you usually like to have a bigger sample...though, that's mostly because you will frequently divide the respondents into subgroups during the analysis (e.g., older vs. younger, casual vs. avid gamer, etc.)

All good, except when you decide to make such a division (older/younger) before the survey is even released, and pre-determine that you're going to ignore all input from one group.

2) Just because you personally haven't been asked to participate in a WotC market research study doesn't mean their research is flawed. If their numbers are accurate, and several million people play RPGs, then the vast majority of those who play have never participated in a market research study for WotC.

After a fashion, we all were or could have been "personally asked" for the 3e research; wasn't the survey contained in an issue of Dragon?

3) Just because their research says something different from what you feel (or, "me and everyone I know") doesn't mean it's flawed, either. Even in studies I've done in which the results were "overwhelming", there's still 10 or 15% of the respondents who hold minority viewpoints.

Fair enough, and sometimes those 10-15% will cluster so that from their viewpoint they seem to be the norm; I suspect that's what's happened in my own gaming experience.

One other note...most companies never share the results of their market research with the public; not even their avid users. They pay a great deal of money for that research, and the information in it represents a competitive advantage. The fact that that information from that one study in 1999 was released (which, I suspect, was done to help convince other publishers to jump on the d20/OGL bandwagon) seems to have set some unrealistic expectations among the gamer community that we're somehow entitled to see this stuff from WotC on a regular basis.

We-ell, we *could* always have a go at trying to replicate the research here, and throw in a pre-3e and post-3e variant to see how if at all peoples' gaming has changed...yes, EnWorld is a limited forum, but it'd be interesting nonetheless.

I'll get started on some polls...

Edit: ...and have now got 3 going, on number of players, number of characters, and length of campaign; trying in each to also compare pre- and post-3e numbers. Is there anything else that's poll-able...levels, maybe, but I can't think how to get anything useful from that...anything else?

Lanefan

 

[jeffh]

All good, except when you decide to make such a division (older/younger) before the survey is even released, and pre-determine that you're going to ignore all input from one group.

I don't think we know when, or even more importantly, why, that decision was made.

 

[Rhuvein]

A wise man once said that there are three kinds of lies: lies, damn lies, and statistics.

Indeed, ahh what was the question?

 

[kenobi65]

I don't think we know when, or even more importantly, why, that decision was made.

Agreed. While researchers may know ahead of time which subgroups they want to look at in their analysis, no researcher is going to include a group of respondents in their study, knowing full well from the start that they have no intention of using the data. That's just a silly accusation.

 

[kenobi65]

After a fashion, we all were or could have been "personally asked" for the 3e research; wasn't the survey contained in an issue of Dragon?

I don't believe so...while they might have replicated some of the questions in the 3E study in a Dragon survey, IIRC, the actual 3E study was conducted using the household panel of a major U.S. market research company.

Using the Dragon readership alone for a study like this would be pretty similar to the effect of just using a group like EN World subscribers...they're a subgroup of the total player base, and probably not a completely representative subset.

 

[Mishihari Lord]

Just out of curiosity, kenobi65, how much does a market study like WOTC's generally cost?

 

[kenobi65]

Just out of curiosity, kenobi65, how much does a market study like WOTC's generally cost?

Easily $75K to $100K, possibly as much as $200K or more, depending on how much analytical work they paid for (e.g., multivariate analyses), and how much of the report writing and project management they did in-house, versus having the research supplier do.

 

[jeffh]

Easily $75K to $100K, possibly as much as $200K or more, depending on how much analytical work they paid for (e.g., multivariate analyses), and how much of the report writing and project management they did in-house, versus having the research supplier do.

Holy crow. I had no conception it cost that kind of money. If I'd had to guess I'd damn near have shaved an entire digit off those figures.

 

[Hussar]

Fair enough, and sometimes those 10-15% will cluster so that from their viewpoint they seem to be the norm; I suspect that's what's happened in my own gaming experience.

And, let's not forget that people will be far more likely to complain than to compliment. If there is a great welling of silence regarding a well selling product, it's because everyone's happy, not that they're ambivalent. But, when a product gets changed, some people aren't happy and are likely going to let everyone know about it.

Something to remember too about the age of gamers - EnWorld is populated by fairly old gamers. So, it makes sense that we got into the game in 1e. A fifteen year old gamer is far more likely to post on the WOTC board before EnWorld. We post here because most of us like d20 material and like to talk to those who developed that material.

I would suspect that an age survey done on WOTC would show a fair bit lower age of gamer.

I could be wrong.