D&D General What elements should D&D keep? forums vs. Reddit

[Parmandur]

You might want to read up a bit about survey design and statistics if you want to continue with this project:

"Lack of representativeness often comes from sampling errors or biases. An example of sampling error would be conducting a survey of how many people eat dairy products by recruiting participants from your local popular vegan café. Another example would be studying the drinking habits of college students, but only sampling from members of fraternities. In these examples, it is easy to see how the characteristics of the samples may potentially bias the results."

What is a Representative Sample?

A representative sample is one that accurately represents, reflects, or “is like” your population.

www.statisticssolutions.com

 

[JEB]

With all due respect, @Parmandur, it doesn't sound like I could do a poll that would meet your standards. But if you're starting a collection to have this done by a professional polling firm, I'm happy to chip in...

 

[Parmandur]

With all due respect, @Parmandur, it doesn't sound like I could do a poll that would meet your standards. But if you're starting a collection to have this done by a professional polling firm, I'm happy to chip in...

Well, yes, a statistically valid forum poll is highly improbable, maybe literally impossible. WotC seems to have the "survey the D&D population" angle covered.

Which is what really makes me curious: granted that this is statistical noise and junk data, what was the point of this exercise? What is the end goal here? None of the results give us information on even the ENworld or D&D Reddit populations, given the response rates and lack of sample controls of any sort, plus ambiguities in the survey design. What's the end game here?

 

[JEB]

Which is what really makes me curious: granted that this is statistical noise and junk data, what was the point of this exercise? What is the end goal here? None of the results give us information on even the ENworld or D&D Reddit populations, given the response rates and lack of sample controls of any sort, plus ambiguities in the survey design. What's the end game here?

You're really upset I even asked this question, aren't you?

 

[Parmandur]

You're really upset I even asked this question, aren't you?

Not really, no. I'm somewhat put off that you fought back against some mild constructive criticism of the methodology from @TheSword among others, which could have helped you with building towards...whatever the point of this was supposed to be?

As you admitted in the previous post, this isn't statistically significant data, so...why?

 

[JEB]

Not really, no. I'm somewhat put off that you fought back against some mild constructive criticism of the methodology from @TheSword among others, which could have helped you with building towards...whatever the point of this was supposed to be?

I did this poll because I was curious what the answers would be. I wanted to see what elements people wanted to keep, after it was pointed out that the previous "feel" poll wouldn't necessarily reflect what people liked to have in the game, as opposed to what seemed to be part of its feel. It was also an attempt to find the strongest common denominators, as well as judging the actual support for certain controversial elements.

The way you ask this question suggests you think I had some secret, nefarious purpose in mind...

 

[JEB]

As you admitted in the previous post, this isn't statistically significant data, so...why?

I admitted no such thing. But clearly the data isn't statistically significant to you. I'm sorry my poll fails to meet your requirements.

 

[tetrasodium]

Just to be clear, I'm not saying this poll is perfect, or that it couldn't have been improved, or that 389 results represent the feelings of every single D&D player on the planet. Those would be silly assertions; even the polls Wizards does themselves don't meet that standard, and they're certainly more comprehensive than my simple little survey.

But I do think the results have meaning for the respondents, and likely reflect the general preferences of the forums I polled. And they may possibly reflect some larger trends in the D&D community, since there were some broadly consistent patterns. Maybe the results are a complete fluke, but I don't think that's likely.

Adding to what @Parmandur put forward about sampling, the wording of a poll question & order they are asked in can also dramatically influence the results for example (continued). Your poll had a single question asking about dozens of things that all assume an improbably high level of system knowledge in every recipient then you tried to use those results to show multiple things. Even the sampling is badly flawed, I answered the poll here & then ignored it when I saw it on reddit because iirc you asked people not to respond to it in different places, doing that however makes the differences between communities of dubious value at best since there is going to be significant overlap between communities and you've taken steps to excise that overlap.

Even the way you group the respondents is very important & reddit vrs enworld vrs whatever is simply not a meaningful distinction to playing or running d&d. For example....

• Are the results of a reasonably designed series of survey questions meaningfully different when asked to players vrs GMs
• Are the results of a reasonably designed series of survey questions meaningfully different when asked to VTT gms/VTT players
• Are the results of a reasonably designed series of survey questions meaningfully different when asked to different regions
• Are the results of a reasonably designed series of survey questions meaningfully different when asked to different regions ages

All of these are things that can be asked with one or two questions each & possibly a survey split for GM/player or VTT/No VTT use to potentially show meaningful differences.


I bolded"The polls Wizards does" because Wotc is almost hilariously bad at polling in the vast majority of their polls. Take this common question "Is X fun" yes/no & how fun is X 1-5. Lets use warlock invocations for X

• Majority say yes it's fun ->awesome power level for the class is perfect lets make more equally powerful invocations to meet their needs
• Majority says it's not fun ->Hmm that's a shame, clearly it needs to be stronger lets make more equally powerful invocations to meet their needs as they advance to gain more invocations
• Majority rank on the low end of 1-5 -> See bob, I told you that the invocations need to be stronger, I made a bunch of new versions that crank these weak threes up to 4 or 5, look at this repelling agonizing blast & think ho cool it will be
• Majority rank on the high end of 1-5 -> See bob, I told you we were on the right track. I made this eldritch lance invocation to bump the range on the cool invocations so it's going to be more regularly in range

A question like that doesn't tell you anything about why the respondent answered the way they answered or even how/if the question is relevant to them... are invocations fun?..

• "I'm the gm.... uhh.. i guess no cause they are kinda OP.. or maybe yes cause my warlock player seems happy to be dialed in around 11. Maybe there are other questions for that"
• "well I felt like the warlock kind of overshadowed the rest of us & noticed a couple players talking about rolling one up. Plus I'm kinda sick of hearing 'lets take a short rest' at the end of every fight so I guess no... well the warlock player seemed to be having fun so maybe yes cause I guess none of those problems are covered by this & figure there must be other questions for that"
• "I'm the warlock pc & felt pretty badass so guess that's a yes... I didn't really like the frustrated looks always asking us to take short rests got so maybe no... I heard a couple other players talking about rolling up warlocks & feel like it might cause too many warlocks & that wouldn't be fun... but there must be other questions for that.."
• Surprise there are no other questions for that

In a lot of cases it would be difficult to deliberately make a worse poll than the ones wotc puts out if you tried. I can only think f a couple exceptions & think both of them were this year

 

[Parmandur]

I did this poll because I was curious what the answers would be. I wanted to see what elements people wanted to keep, after it was pointed out that the previous "feel" poll wouldn't necessarily reflect what people liked to have in the game, as opposed to what seemed to be part of its feel. It was also an attempt to find the strongest common denominators, as well as judging the actual support for certain controversial elements.

The way you ask this question suggests you think I had some secret, nefarious purpose in mind...

By no means am I suggesting a nefarious motive, I'm just honestly puzzled as to why you put in the amount of effort you have done, when the results are not even valid data?

I admitted no such thing. But clearly the data isn't statistically significant to you. I'm sorry my poll fails to meet your requirements.

You stated that "it doesn't sound like I could do a poll that would meet your standards." But what I am talking about are minimal standards of statistics as a science, not personal feelings, or some maximal high standard. If the minimum thresholds of data integrity cannot be met...then the data isn't valid.

 

[Parmandur]

Adding to what @Parmandur put forward about sampling, the wording of a poll question & order they are asked in can also dramatically influence the results for example (continued). Your poll had a single question asking about dozens of things that all assume an improbably high level of system knowledge in every recipient then you tried to use those results to show multiple things. Even the sampling is badly flawed, I answered the poll here & then ignored it when I saw it on reddit because iirc you asked people not to respond to it in different places, doing that however makes the differences between communities of dubious value at best since there is going to be significant overlap between communities and you've taken steps to excise that overlap.

Even the way you group the respondents is very important & reddit vrs enworld vrs whatever is simply not a meaningful distinction to playing or running d&d. For example....

• Are the results of a reasonably designed series of survey questions meaningfully different when asked to players vrs GMs
• Are the results of a reasonably designed series of survey questions meaningfully different when asked to VTT gms/VTT players
• Are the results of a reasonably designed series of survey questions meaningfully different when asked to different regions
• Are the results of a reasonably designed series of survey questions meaningfully different when asked to different regions ages

All of these are things that can be asked with one or two questions each & possibly a survey split for GM/player or VTT/No VTT use to potentially show meaningful differences.


I bolded"The polls Wizards does" because Wotc is almost hilariously bad at polling in the vast majority of their polls. Take this common question "Is X fun" yes/no & how fun is X 1-5. Lets use warlock invocations for X

• Majority say yes it's fun ->awesome power level for the class is perfect lets make more equally powerful invocations to meet their needs
• Majority says it's not fun ->Hmm that's a shame, clearly it needs to be stronger lets make more equally powerful invocations to meet their needs as they advance to gain more invocations
• Majority rank on the low end of 1-5 -> See bob, I told you that the invocations need to be stronger, I made a bunch of new versions that crank these weak threes up to 4 or 5, look at this repelling agonizing blast & think ho cool it will be
• Majority rank on the high end of 1-5 -> See bob, I told you we were on the right track. I made this eldritch lance invocation to bump the range on the cool invocations so it's going to be more regularly in range

A question like that doesn't tell you anything about why the respondent answered the way they answered or even how/if the question is relevant to them... are invocations fun?..

• "I'm the gm.... uhh.. i guess no cause they are kinda OP.. or maybe yes cause my warlock player seems happy to be dialed in around 11. Maybe there are other questions for that"
• "well I felt like the warlock kind of overshadowed the rest of us & noticed a couple players talking about rolling one up. Plus I'm kinda sick of hearing 'lets take a short rest' at the end of every fight so I guess no... well the warlock player seemed to be having fun so maybe yes cause I guess none of those problems are covered by this & figure there must be other questions for that"
• "I'm the warlock pc & felt pretty badass so guess that's a yes... I didn't really like the frustrated looks always asking us to take short rests got so maybe no... I heard a couple other players talking about rolling up warlocks & feel like it might cause too many warlocks & that wouldn't be fun... but there must be other questions for that.."
• Surprise there are no other questions for that

In a lot of cases it would be difficult to deliberately make a worse poll than the ones wotc puts out if you tried. I can only think f a couple exceptions & think both of them were this year

There's a certain misunderstanding of WotC intentions with UA polling there: UA is not for balance testing, but for market desirability testing. They can balance for power internally using their playtesting tools and math. UA is all about determining if people actually want an option at all. And they have started asking questions to sort out responses such as "What Edition did you start playing?", "Are you a DM?", "Were you interested in this sort of concept before you read this article, "Are you more likely to want to play this type after reading the article?"