Interesting Ryan Dancey comment on "lite" RPGs

[Steve Conan Trustrum]

Not a bad guess, though - the article from 2000 said 1.5 million gamers in the U.S.

That's nothing more than a guess on their part, however. The truth is nobody has any idea how many gamers there are.

 

[WizarDru]

He was tossing out a personal observation from the role of someone who was there in the first place in the capacity of someone who was meant to draw a conclusion about the research that would then influence the product line.

Fair enough, but my point was that he was tossing it out as an anecdote only tangential to the discussion of lite-gaming systems, and I don't know how well thought out it was. I'm sure it wasn't intended to withstand months of geek scrutiny.

research, even just looking at what little Ryan revealed of it, isn't sufficient to the conclusions he made about what he saw. The same can also be said of their well-known survey results. Again, the data collected wasn't sufficient to make the conclusions they presented valid.

Well, I can't argue either way about the validity of their data, but I'm certainly willing to accept that they looked to have the data validate their ideas, and entered into it, looking for the data to provide them the answer they already had decided upon. I've certainly encountered that mindset at a certain Government Contractor who shall remain Nameless.

The ability to spend money on research doesn't mean the results are accurate or the research was conducted properly. I've worked with some companies a lot larger than WotC (a few that are larger than Hasbro) and their "well planned" research projects were a joke.

As I said, I concede the point. What I was trying to point out, though, was that only a handful of companies in the RPG or games industry can even afford to do such research in the first place. That certainly makes it difficult to do lots of comparisons or analyses, as I think we're seeing here.

Here's a question: how important is it to have that data, for WotC or others? Clearly, lack of this kind of information killed TSR...but that was one bad practice among many. I mean, Green Ronin and Malhavoc keep in contact with their fan bases, for example, and their print runs are still tiny compared to WotC...do they gain enough benefit to care, from such research? Is there an industry organization that has any strength and resources to do a more unbiased review?

 

[Patryn of Elvenshae]

Not with large demographics, to be certain, but such polls do indeed exist.

And there you have it.

The only way a full-demographic poll works is if you're targeting a group of less than a hundred members or so.*

Seriously, though, are you just being obtuse, or do you not recognize hyperbole? Calling their methods into question because they used the most widely-accepted method of polling is coming it a little high.

Joe Average? I'm not here to compare resumes, bub.

* Beyond that, you're going to need some form of mandatory polling, and some serious weight to back up your desires. For instance, you could probably (with some time) get a 100% sample rate out of a military unit. It would be nigh-impossible in any "real world" application, however.

 

[Steve Conan Trustrum]

And there you have it.

Yes, there indeed is the statement that refutes your claim. Please remember, you didn't qualify your assertation along the lines of "most commonly used" (although I have myself addressed this.) What you said was every other poll in the history of statistics has every been done. Now, this clearly isn't true. If you want to say that complete polling isn't common, you'd have a different point and I'd agree with you. That is not, however, at all what you said.

The only way a full-demographic poll works is if you're targeting a group of less than a hundred members or so.*

Again, not true. As perhaps the most common example that comes to mind, we do a lot of medical research. We have found ourselves polling all of a given medical specialty in Canada. On occassion we get them 100% completed, although all come close because the specialists have a vested interest in answering. These polls are done with over 100 respondents. This is done by arranging in-office interviews, scheduling times to call rather than blanket calling, mailouts with a long window of returns, etc. EDIT: All these methods still use the same standardized, quantitative poll but recognize that just calling people up and hoping for the best won't get it done. Again, it's not common but it certainly does occur despite your claims to the contrary. It's not like we tell a client "what, you want to ask 101 people this survey? Well, I'm sorry, but the best we'll be able to do is get 100 replies for you." The greater the sample, the less the chance of a 100% return, but it ISpossible.

Seriously, though, are you just being obtuse, or do you not recognize hyperbole? Calling their methods into question because they used the most widely-accepted method of polling is coming it a little high.

Obtuse? Actually, I'm being very direct.

You'll also note I'm not saying that it was their polling that comes into question (I don't have any information on their call plan to make that comment), so much as the conclusions they drew from it. There are some conclusions in there (such as the # of gamers in the US) that simply cannot be arrived at with accuracy from such a poll--certainly not one instance of such a poll. To even arrive at a reasonable estimate they would have to conduct a longitudinal poll of shifting demographics to see if patterns of consistency showed up throughout the various regions. Without doing so extrapolating a nationwide number is an guess the data simply can't support truthfully.

Joe Average? I'm not here to compare resumes, bub.

That much, sir, is obvious.

It would be nigh-impossible in any "real world" application, however.

Say with, oh, I don't know, medical specialists?

 

[Patryn of Elvenshae]

Yes, there indeed is the statement that refutes your claim. Please remember, you didn't qualify your assertation along the lines of "most commonly used" (although I have myself addressed this.) What you said was every other poll in the history of statistics has every been done.

In other words, you're having trouble with your English comprehension skills. Specifically, you can't understand hyperbole. That's ok - I'm sure your ex-English teachers will forgive you.

We have found ourselves polling all of a given medical specialty in Canada. On occassion we get them 100% completed, although all come close because the specialists have a vested interest in answering. These polls are done with over 100 respondents.

I've highlighted the important parts, and reintroduced a statement I've already made:

* Beyond that, you're going to need some form of mandatory polling, and some serious weight to back up your desires.

Unless I miss my guess, healthcare in Canada is socialized and is therefore, at least in part, state-run. Would this be a correct statement?

So, what group size are we talking about here? Larger than 100, you say, but then you also mention that you only occasionally get a 100% sample rate (and occasionally may mean "Once, like three years ago").

I'm currently party to a survey sent to 300 respondents (actually, three separate surveys sent to three groups of respondents of varying size). Every respondent has a highly-vested interest in replying - the amount of money they'll be getting from the business who requested the survey is highly dependent upon their answers. I will be shocked and amazed if anything like a 50% rate of return is achieved, even with the phone and e-mail follow-ups we have planned.

 

[Steve Conan Trustrum]

In other words, you're having trouble with your English comprehension skills. Specifically, you can't understand hyperbole. That's ok - I'm sure your ex-English teachers will forgive you.

Something only counts as hyperbole if you can illustrate how you intend it to be an exaggeration. Your comments so far have done very little to convince me you are a competent researcher, so expecting me to think you're using hyperbole rather than speaking from lay knowledge (especially given the context in which you used it) is a leap that grants you more merit than your words warrant thus far.

I've highlighted the important parts, and reintroduced a statement I've already made:

And, again, how did I suggest otherwise? You were talking in extremes and absolutes. I was not. If you want to account for exceptions--no matter how they are formed--please stop speaking in unqualified absolutes and then assume that you have already done so when called on it. You certainly have not done so, sir, unless you are again going to tell me the following unqualified statement is hyperbole: The only way a full-demographic poll works is if you're targeting a group of less than a hundred members or so. If you did mean this to be taken at face value, I believe what you meant to say was The only way a GENERAL POPULATION full-demographic poll works is if you're targeting a group of less than a hundred members or so (which is an interesting concept in and of itself considering the idea of a survey targetting a sample of 100 that isn't going after a niche is rather odd, but I'll give you the benefit of the doubt), but you did not say that, nor did you so much as suggest it. Please, do start saying what you mean rather than trying to add unspecified post-statement criteria to your points and my comments.

Honestly, you jump on one point in the post with what you claim is hyperbole, go back and add unspecified criteria to your comments afterward to skew my replies, and yet failed to address ANY of the other points about the research I made. So, instead of chipping away at one corner of the stone with your tiny hammer, how about addressing your expertise to the rest of the points I made with something other than claiming an observer at a market research focus group is only offering his "personal opinion," as though he were present in a social capacity, when he clearly is not. Please speak to the validity of the comments rather than excusing the points they address with a clearly apologetic and thin excuse.

Unless I miss my guess, healthcare in Canada is socialized and is therefore, at least in part, state-run. Would this be a correct statement?

So, what group size are we talking about here? Larger than 100, you say, but then you also mention that you only occasionally get a 100% sample rate (and occasionally may mean "Once, like three years ago").

Research participation isn't subsidized to the individual healthcare workers in Canada. Also, much of our healthcare research is for American pharmaceutical companies.

The size of the group depends on the specialty and how we contact them. A touch on behalf of Roche, for example, certainly won't garner the same response as a charity. Of course, if you're in research you'll know this already.

And, AGAIN, yes "occassionally" can indeed mean once in three years. It can mean once in a decade. But, as I've pointed out, I was not the one who put a time frame on how often this occurs. I merely stated that it CAN happen. I stated it was not impossible in spite of your claim otherwi... er, in spite of your "hyperbole." If you wish to, in hindsight, try to claim I made a claim to the contrary, I beg you to point it out so that I can apologize for the unclear communication of my point. If not, I once more suggest you stick to what you and I are both saying rather than adding imaginative implications to my words post hoc.

I'm currently party to a survey sent to 300 respondents (actually, three separate surveys sent to three groups of respondents of varying size). Every respondent has a highly-vested interest in replying - the amount of money they'll be getting from the business who requested the survey is highly dependent upon their answers. I will be shocked and amazed if anything like a 50% rate of return is achieved, even with the phone and e-mail follow-ups we have planned.

Perhaps you should farm the survey off to us, then.

The first thing I'd point out to you and your client is that making the incentive varied to suit responses will automatically skew the data instead of offering a flat fee for qualified participation to represent an unbiased standard. An incentive is meant to be a flat payment for services rendered, not a reward for giving the "correct" answers--doing the latter is definately leading your respondent. But, seeing as how you are CLEARLY a market researcher I need not point that out to you ... Having dealt with that issue, I'm certain we could offer suggestions on getting your return rate up.

We both stray far from the mark, however. Do you have any suggestions on how WotC--indeed, any rpg company--could get reliable market research data at a cost most companies in this industry can afford?

 

[Patryn of Elvenshae]

Something only counts as hyperbole if you can illustrate how you intend it to be an exaggeration.

And you haven't been able to pick up on that, yet?

 

[Steve Conan Trustrum]

And you haven't been able to pick up on that, yet?

Just waiting for you to act on it. In a textual communication medium like email or a messageboard, I realize that sarcasm and hyperbole cannot be taken for granted. Seeing as I've no history of communication with you by which to judge your responses and penchant for hindsight reconstruction, what basis do I have to take what you say for granted in any way other than literally?

Now, if you're quite through petualantly trying to do the latter, why not address the other points I brought up? What about the requirement of longitudinal tracking? Better yet, what would you suggest to make market research more accessible to gaming companies without sacrificing quality? (In fact, this thread has got me thinking about this rather seriously.)

 

[Turjan]

The only way a full-demographic poll works is if you're targeting a group of less than a hundred members or so.*

* Beyond that, you're going to need some form of mandatory polling, and some serious weight to back up your desires. For instance, you could probably (with some time) get a 100% sample rate out of a military unit. It would be nigh-impossible in any "real world" application, however.

I had to take part in one mandatory poll in my home country. Goal was a full-demographic poll with 100% return. That were roughly 62 million poll participants. The way they did it, it was unavoidable for most people, and they also checked whether you wrote rubbish. That were a few more people than 100 .

 

[Patryn of Elvenshae]

I had to take part in one mandatory poll in my home country.

Which is exactly the kind of enforcement I'm arguing is going to be required if you want a 100% return rate on any sizeable demographic.

EDIT: BTW, what is your home country, and what was the poll? I'd imagine it was a Census of some form (which, at least in the US, uses the same system of extrapolation that "Conan" is disparaging).