D&D 5E D&DN going down the wrong path for everyone.

[AbdulAlhazred]

Thank you both.

I am not a business manager, nor an academic, but the idea that WotC's survey data is worthless is one I can't take seriously. The idea that the gaming practices might be radically different from what their data suggest I find pretty unlikely. The idea that any significant number of those 20-session campaigns started at a level much above 3rd I also find pretty doubtful, but that last point is my own intuition and not itself inherent in the example.

Yeah, I have to say, such suggestions strike me as simply people refusing to accept how things really are. I've VERY rarely started past level one or participated in a campaign where that was a practice, etc. The fact that the survey data matches my own observations over more than 3 decades of RPing just gives that last added ring of authenticity.

Beyond that, if WotC is concentrating mostly on low-mid level play, and that seems to be what they've consistently done for a long time with very limited exceptions then it stands to reason THEY believe the data is good. No doubt they've continued to collect data over the last 15 years and have a pretty good handle on how people play. It really just isn't that hard, even a small market data research firm can do a study like that for small money.

 

[Obryn]

THANK YOU! This should be stickied in a locked thread at the top of the forums. It's impossible to have a reasonable discussion about gamer preferences when folks insist on throwing out data they don't agree with because the research methodology isn't perfect. No market research is perfect, everything has bias. Be aware of the biases and imperfections, but even with these you can still learn a lot from imperfect data. As a business manager, I have to make decisions all of the time based on imperfect data - you just have to do the best you can. Decisions involving issues quite a lot of money are made every day with less certain data than some cited in this thread.

Excellent point. I think we're seeing a difference of "academic"/"math geek" discussions of significance versus "marketing"/"business" significance.

While in a perfect world these would match, they just don't in practice. Because your customers are not experimental participants, and there's no such thing as clean data.

-O

 

[billd91]

Honestly though: I had a job number-crunching for a fairly sophisticated market research firm, and I hold a math degree. Mostly people oversell this "you can't tell anything from the data, its all just lying statistics/bad sampling/etc" meme. The fact is that the sample sigma will approach the population sigma for a sample size N of 10 (assuming that the population size is MUCH larger than N). Surveying a representative handful of gamers (several hundred) should, with reasonably competent study implementation, give you a quite good idea of what the population is like. Exactly HOW good? That depends to some extent on the nature of the population and how heterogeneous it is, how carefully the sample's biases are understood and corrected for, etc. I'd note that there are perfectly reasonable ways to TEST how well a sample matches a population as well, internal sub-sampling analysis, additional test samples, etc. The truth is marketing organizations largely can't exist if they can't analyze their effectiveness. While its a wonderful theory to just call any marketing study you don't like 'incompetent' the truth is that's not usually the case. The techniques are quite well-understood at this point, and these groups will almost certainly have a statistician on staff who can do the required analysis and design studies properly.

So would you call a sample in which they deliberately and arbitrarily decided to exclude responses from 35+ year olds unbiased? And are you going to say that the summary posted by Sean K. Reynolds supports all the assertions people are making here in this thread? Or are you going to continue to defend both of those because it fits your preconceived notions?

Not really. Again, just because you SAY some people played very high level play doesn't make it so. No doubt SOME people did, but the VAST majority of all characters ever run are low level PCs for which level caps would never ever matter. That seems quite in keeping with my experience.

And just because you say it's not in keeping with your experience doesn't say much either.

A good staff or a kick assed sword is worth more than several levels of post-name-level advancement.

They're certainly worth more than a few levels of pre-name-level advancement. But I'd not discount the higher level spells of the wizard or the impressive saving throws of the high level fighter.

 

[ForeverSlayer]

Thanks for bringing an iota of sense to this utterly barmy thread. The idea that "only a small proportion of the total population replied" is a reason for non-representativeness of surveys is an ignorant misconception. Any sample of around 60+ can have average values that follow a normal distribution (and are thus representative and susceptible to statistical tests for validity). The reason the surveys might be biased are because of sampling methodologies and hypothetical questions, not because of low sample size.

Oh, and the "95% of all women" thing might even be from a representative sample, but they don't cite the hypothesis confidence level, which is usually the killer datum.

Make some sense in this thread? Are you honestly going to sit there and fall into the fallacy of the minority speaks for the majority nonsense? There is nothing wrong with data, but you have to be careful how you use it and how you are using it to back you up. If you have 32% of the population of a country actually go out and vote in the next election then you can't honestly say that the country wanted the particular political party into power because of such a low turn out but you have to count the votes that are actually there. It's freaking statistics 101.

Darcey's survey doesn't represent the majority of players nor does it represent the minority, we don't know who it represents because we can't actually go out and do a head count. People who are arguing this still really need to go and do some study on statistics. Games may have fallen short amongst the people that were actually asked but like Bill said, a portion of the gaming community was left out so thar automatically makes the survey flawed.

Some of you are grasping for straws so desperately that you will hold on to anything you can get.

 

[Balesir]

Make some sense in this thread? Are you honestly going to sit there and fall into the fallacy of the minority speaks for the majority nonsense?

Under specified conditions, it's not a fallacy. Any independently and identically distributed (i.i.d. - yes, that is a technical term) sample of reasonable size (around 50-60+) can be assumed to be representative of the population it is taken from (with larger samples generally giving progressively narrower confidence intervals in the results). Of course, getting it to be i.i.d. is the hard part.

There is nothing wrong with data, but you have to be careful how you use it and how you are using it to back you up. If you have 32% of the population of a country actually go out and vote in the next election then you can't honestly say that the country wanted the particular political party into power because of such a low turn out but you have to count the votes that are actually there.

And, as anyone who has studied statistics will tell you, that's not because of the small sample size, it's because the sample is self-selecting. If 32% (or even less) of the population were chosen by genuinely random lottery then their voting would be quite representative of the voting of the population as a whole. Polling companies try to do just this to predict election outcomes, but they have hypothetical question bias to contend with as well as sampling issues, making the whole process more than a bit challenging...

It's freaking statistics 101.

Actually, 'Statistics 101' will generally go at least as far as I've explained above, so it's already a good deal more nuanced than you are making it out to be.

Darcey's survey doesn't represent the majority of players nor does it represent the minority, we don't know who it represents because we can't actually go out and do a head count.

I don't know who Darcey is, but Ryan Dancey's data I would assume to be tolerably representative of the population of "gamers under 35 at the time the data was collected". The might not be the dataset we would ideally like, but it's at least something (which is more than you have put forward).

People who are arguing this still really need to go and do some study on statistics.

I have just recently completed a Master's degree course in Econometrics (statistical methods for analysing economic and financial data); this is enough to show me that your claims on statistics so far are at best selective and incomplete.

Games may have fallen short amongst the people that were actually asked but like Bill said, a portion of the gaming community was left out so thar automatically makes the survey flawed.

No, it makes it a survey of the population it says it is a survey of. Would it be better for the purposes of this discussion if it was a survey representative of all gamers? Absolutely! But it's not, so absent anyone volunteering to fund a new survey tailored to our desires, it's what we have.

Some of you are grasping for straws so desperately that you will hold on to anything you can get.

I'm not particularly grasping for anything (except a discussion that is at least minimally informed), but if the cap fits feel free to wear it.

 

[UngeheuerLich]

Sorry forever slayer, but balesir explains it quite well. Actually the study has asked so many people, that it is better than the average study.
The only possible bias i can see, is that maybe people who are gaming are more likely to send their letters back, because they have a greter interest in voicing their opinion and preferences.
[MENTION=36799]Wom[/MENTION]en asked: it is not the number of women asked but the kind of. I expect 95% of 250 women happy with the product was gathered by a person who aked the woman after she bought the product (why would you buy it, if you are not happy with the product.) Even when asking more neutral women... it is quite impossible, that some products make you unhappy. So a binary poll could easily end up in such great percentages. (Ok - Not OK poll Interpreted as whoever chose ok is happy)

 

[ForeverSlayer]

Sorry forever slayer, but balesir explains it quite well. Actually the study has asked so many people, that it is better than the average study.
The only possible bias i can see, is that maybe people who are gaming are more likely to send their letters back, because they have a greter interest in voicing their opinion and preferences.
[MENTION=36799]Wom[/MENTION]en asked: it is not the number of women asked but the kind of. I expect 95% of 250 women happy with the product was gathered by a person who aked the woman after she bought the product (why would you buy it, if you are not happy with the product.) Even when asking more neutral women... it is quite impossible, that some products make you unhappy. So a binary poll could easily end up in such great percentages. (Ok - Not OK poll Interpreted as whoever chose ok is happy)

It could have been a first time use study. You can't technically purchase only things you like for the first time, you have to try out the product first before you decide you like it.

 

[ForeverSlayer]

Under specified conditions, it's not a fallacy. Any independently and identically distributed (i.i.d. - yes, that is a technical term) sample of reasonable size (around 50-60+) can be assumed to be representative of the population it is taken from (with larger samples generally giving progressively narrower confidence intervals in the results). Of course, getting it to be i.i.d. is the hard part.

And, as anyone who has studied statistics will tell you, that's not because of the small sample size, it's because the sample is self-selecting. If 32% (or even less) of the population were chosen by genuinely random lottery then their voting would be quite representative of the voting of the population as a whole. Polling companies try to do just this to predict election outcomes, but they have hypothetical question bias to contend with as well as sampling issues, making the whole process more than a bit challenging...

Actually, 'Statistics 101' will generally go at least as far as I've explained above, so it's already a good deal more nuanced than you are making it out to be.

I don't know who Darcey is, but Ryan Dancey's data I would assume to be tolerably representative of the population of "gamers under 35 at the time the data was collected". The might not be the dataset we would ideally like, but it's at least something (which is more than you have put forward).

I have just recently completed a Master's degree course in Econometrics (statistical methods for analysing economic and financial data); this is enough to show me that your claims on statistics so far are at best selective and incomplete.

No, it makes it a survey of the population it says it is a survey of. Would it be better for the purposes of this discussion if it was a survey representative of all gamers? Absolutely! But it's not, so absent anyone volunteering to fund a new survey tailored to our desires, it's what we have.

I'm not particularly grasping for anything (except a discussion that is at least minimally informed), but if the cap fits feel free to wear it.

I'm sorry but we are just going to have to agree to disagree.

The survey is not a reliable enough source to back you up as factual information.

 

[Jester David]

So would you call a sample in which they deliberately and arbitrarily decided to exclude responses from 35+ year olds unbiased?

Keep in mind this survey is also 14 years old and was likely launched as part of the build up to 3e. What this does mean is that D&D was only 25 years old. So someone who got into the game at age 15 at the peak in 1981 would be around 33.
So aiming it at 12-35 year olds makes sense. It's going to have a slight bias but not remarkably so, given the 36+ bracket would be a smaller audience and one likely to be even busier and gaming less frequently (and also buying fewer books and less likely to change editions which makes them less of a marketable audience). And they'd likely skew the average start age statistics.

Looking at the statistics, it looks like they cast a wide net so it should be fairly representational. They have some good numbers. Might not be entirely accurate, but it just as easily might be accurate. And it's the only data we have, so unless you're willing to offer a better survey it should be cautiously relied on.

Really, the best reason to be wary about the data is that it's 14 years old. The demographics of the game have likely changed remarkably. A lot of the numbers might be off for that reason.

 

[Ahnehnois]

I'm sorry but we are just going to have to agree to disagree.

The survey is not a reliable enough source to back you up as factual information.

I agree. While it's nice to think that WotC at one point made an effort to find out what their customers wanted, it would be absurd to think that they reached a representative sample of players. I mean, how would you find them? How many D&D players participate in any kind of organized play, conventions, or even go to specialty stores to play their games? A vanishingly small number, I think. Moreover, D&D is a stigmatized hobby, not something that many people are likely to talk about even if asked. And this was likely more true fifteen years ago than it is now.

I don't see any realistic mechanism for locating a group defined by a hobby they participate in by buying a relatively small number of products from diverse vendors and use at home in private with small groups of people.