• COMING SOON! -- Level Up: Advanced 5th Edition! Level up your 5E game! The standalone advanced 5E tabletop RPG adds depth and diversity to the game you love!
log in or register to remove this ad

 

D&D General What elements should D&D keep? forums vs. Reddit

Parmandur

Book-Friend
You might want to read up a bit about survey design and statistics if you want to continue with this project:

"Lack of representativeness often comes from sampling errors or biases. An example of sampling error would be conducting a survey of how many people eat dairy products by recruiting participants from your local popular vegan café. Another example would be studying the drinking habits of college students, but only sampling from members of fraternities. In these examples, it is easy to see how the characteristics of the samples may potentially bias the results."

 

log in or register to remove this ad

JEB

Hero
With all due respect, @Parmandur, it doesn't sound like I could do a poll that would meet your standards. But if you're starting a collection to have this done by a professional polling firm, I'm happy to chip in...
 

Parmandur

Book-Friend
With all due respect, @Parmandur, it doesn't sound like I could do a poll that would meet your standards. But if you're starting a collection to have this done by a professional polling firm, I'm happy to chip in...
Well, yes, a statistically valid forum poll is highly improbable, maybe literally impossible. WotC seems to have the "survey the D&D population" angle covered.

Which is what really makes me curious: granted that this is statistical noise and junk data, what was the point of this exercise? What is the end goal here? None of the results give us information on even the ENworld or D&D Reddit populations, given the response rates and lack of sample controls of any sort, plus ambiguities in the survey design. What's the end game here?
 

JEB

Hero
Which is what really makes me curious: granted that this is statistical noise and junk data, what was the point of this exercise? What is the end goal here? None of the results give us information on even the ENworld or D&D Reddit populations, given the response rates and lack of sample controls of any sort, plus ambiguities in the survey design. What's the end game here?
You're really upset I even asked this question, aren't you?
 

Parmandur

Book-Friend
You're really upset I even asked this question, aren't you?
Not really, no. I'm somewhat put off that you fought back against some mild constructive criticism of the methodology from @TheSword among others, which could have helped you with building towards...whatever the point of this was supposed to be?

As you admitted in the previous post, this isn't statistically significant data, so...why?
 

JEB

Hero
Not really, no. I'm somewhat put off that you fought back against some mild constructive criticism of the methodology from @TheSword among others, which could have helped you with building towards...whatever the point of this was supposed to be?
I did this poll because I was curious what the answers would be. I wanted to see what elements people wanted to keep, after it was pointed out that the previous "feel" poll wouldn't necessarily reflect what people liked to have in the game, as opposed to what seemed to be part of its feel. It was also an attempt to find the strongest common denominators, as well as judging the actual support for certain controversial elements.

The way you ask this question suggests you think I had some secret, nefarious purpose in mind...
 


Just to be clear, I'm not saying this poll is perfect, or that it couldn't have been improved, or that 389 results represent the feelings of every single D&D player on the planet. Those would be silly assertions; even the polls Wizards does themselves don't meet that standard, and they're certainly more comprehensive than my simple little survey.

But I do think the results have meaning for the respondents, and likely reflect the general preferences of the forums I polled. And they may possibly reflect some larger trends in the D&D community, since there were some broadly consistent patterns. Maybe the results are a complete fluke, but I don't think that's likely.
Adding to what @Parmandur put forward about sampling, the wording of a poll question & order they are asked in can also dramatically influence the results for example (continued). Your poll had a single question asking about dozens of things that all assume an improbably high level of system knowledge in every recipient then you tried to use those results to show multiple things. Even the sampling is badly flawed, I answered the poll here & then ignored it when I saw it on reddit because iirc you asked people not to respond to it in different places, doing that however makes the differences between communities of dubious value at best since there is going to be significant overlap between communities and you've taken steps to excise that overlap.

Even the way you group the respondents is very important & reddit vrs enworld vrs whatever is simply not a meaningful distinction to playing or running d&d. For example....
  • Are the results of a reasonably designed series of survey questions meaningfully different when asked to players vrs GMs
  • Are the results of a reasonably designed series of survey questions meaningfully different when asked to VTT gms/VTT players
  • Are the results of a reasonably designed series of survey questions meaningfully different when asked to different regions
  • Are the results of a reasonably designed series of survey questions meaningfully different when asked to different regions ages
All of these are things that can be asked with one or two questions each & possibly a survey split for GM/player or VTT/No VTT use to potentially show meaningful differences.


I bolded"The polls Wizards does" because Wotc is almost hilariously bad at polling in the vast majority of their polls. Take this common question "Is X fun" yes/no & how fun is X 1-5. Lets use warlock invocations for X
  • Majority say yes it's fun ->awesome power level for the class is perfect lets make more equally powerful invocations to meet their needs
  • Majority says it's not fun ->Hmm that's a shame, clearly it needs to be stronger lets make more equally powerful invocations to meet their needs as they advance to gain more invocations
  • Majority rank on the low end of 1-5 -> See bob, I told you that the invocations need to be stronger, I made a bunch of new versions that crank these weak threes up to 4 or 5, look at this repelling agonizing blast & think ho cool it will be
  • Majority rank on the high end of 1-5 -> See bob, I told you we were on the right track. I made this eldritch lance invocation to bump the range on the cool invocations so it's going to be more regularly in range
A question like that doesn't tell you anything about why the respondent answered the way they answered or even how/if the question is relevant to them... are invocations fun?..
  • "I'm the gm.... uhh.. i guess no cause they are kinda OP.. or maybe yes cause my warlock player seems happy to be dialed in around 11. Maybe there are other questions for that"
  • "well I felt like the warlock kind of overshadowed the rest of us & noticed a couple players talking about rolling one up. Plus I'm kinda sick of hearing 'lets take a short rest' at the end of every fight so I guess no... well the warlock player seemed to be having fun so maybe yes cause I guess none of those problems are covered by this & figure there must be other questions for that"
  • "I'm the warlock pc & felt pretty badass so guess that's a yes... I didn't really like the frustrated looks always asking us to take short rests got so maybe no... I heard a couple other players talking about rolling up warlocks & feel like it might cause too many warlocks & that wouldn't be fun... but there must be other questions for that.."
  • Surprise there are no other questions for that
In a lot of cases it would be difficult to deliberately make a worse poll than the ones wotc puts out if you tried. I can only think f a couple exceptions & think both of them were this year
 

Parmandur

Book-Friend
I did this poll because I was curious what the answers would be. I wanted to see what elements people wanted to keep, after it was pointed out that the previous "feel" poll wouldn't necessarily reflect what people liked to have in the game, as opposed to what seemed to be part of its feel. It was also an attempt to find the strongest common denominators, as well as judging the actual support for certain controversial elements.

The way you ask this question suggests you think I had some secret, nefarious purpose in mind...

By no means am I suggesting a nefarious motive, I'm just honestly puzzled as to why you put in the amount of effort you have done, when the results are not even valid data?

I admitted no such thing. But clearly the data isn't statistically significant to you. I'm sorry my poll fails to meet your requirements.
You stated that "it doesn't sound like I could do a poll that would meet your standards." But what I am talking about are minimal standards of statistics as a science, not personal feelings, or some maximal high standard. If the minimum thresholds of data integrity cannot be met...then the data isn't valid.
 

Parmandur

Book-Friend
Adding to what @Parmandur put forward about sampling, the wording of a poll question & order they are asked in can also dramatically influence the results for example (continued). Your poll had a single question asking about dozens of things that all assume an improbably high level of system knowledge in every recipient then you tried to use those results to show multiple things. Even the sampling is badly flawed, I answered the poll here & then ignored it when I saw it on reddit because iirc you asked people not to respond to it in different places, doing that however makes the differences between communities of dubious value at best since there is going to be significant overlap between communities and you've taken steps to excise that overlap.

Even the way you group the respondents is very important & reddit vrs enworld vrs whatever is simply not a meaningful distinction to playing or running d&d. For example....
  • Are the results of a reasonably designed series of survey questions meaningfully different when asked to players vrs GMs
  • Are the results of a reasonably designed series of survey questions meaningfully different when asked to VTT gms/VTT players
  • Are the results of a reasonably designed series of survey questions meaningfully different when asked to different regions
  • Are the results of a reasonably designed series of survey questions meaningfully different when asked to different regions ages
All of these are things that can be asked with one or two questions each & possibly a survey split for GM/player or VTT/No VTT use to potentially show meaningful differences.


I bolded"The polls Wizards does" because Wotc is almost hilariously bad at polling in the vast majority of their polls. Take this common question "Is X fun" yes/no & how fun is X 1-5. Lets use warlock invocations for X
  • Majority say yes it's fun ->awesome power level for the class is perfect lets make more equally powerful invocations to meet their needs
  • Majority says it's not fun ->Hmm that's a shame, clearly it needs to be stronger lets make more equally powerful invocations to meet their needs as they advance to gain more invocations
  • Majority rank on the low end of 1-5 -> See bob, I told you that the invocations need to be stronger, I made a bunch of new versions that crank these weak threes up to 4 or 5, look at this repelling agonizing blast & think ho cool it will be
  • Majority rank on the high end of 1-5 -> See bob, I told you we were on the right track. I made this eldritch lance invocation to bump the range on the cool invocations so it's going to be more regularly in range
A question like that doesn't tell you anything about why the respondent answered the way they answered or even how/if the question is relevant to them... are invocations fun?..
  • "I'm the gm.... uhh.. i guess no cause they are kinda OP.. or maybe yes cause my warlock player seems happy to be dialed in around 11. Maybe there are other questions for that"
  • "well I felt like the warlock kind of overshadowed the rest of us & noticed a couple players talking about rolling one up. Plus I'm kinda sick of hearing 'lets take a short rest' at the end of every fight so I guess no... well the warlock player seemed to be having fun so maybe yes cause I guess none of those problems are covered by this & figure there must be other questions for that"
  • "I'm the warlock pc & felt pretty badass so guess that's a yes... I didn't really like the frustrated looks always asking us to take short rests got so maybe no... I heard a couple other players talking about rolling up warlocks & feel like it might cause too many warlocks & that wouldn't be fun... but there must be other questions for that.."
  • Surprise there are no other questions for that
In a lot of cases it would be difficult to deliberately make a worse poll than the ones wotc puts out if you tried. I can only think f a couple exceptions & think both of them were this year
There's a certain misunderstanding of WotC intentions with UA polling there: UA is not for balance testing, but for market desirability testing. They can balance for power internally using their playtesting tools and math. UA is all about determining if people actually want an option at all. And they have started asking questions to sort out responses such as "What Edition did you start playing?", "Are you a DM?", "Were you interested in this sort of concept before you read this article, "Are you more likely to want to play this type after reading the article?"
 

JEB

Hero
Even the sampling is badly flawed, I answered the poll here & then ignored it when I saw it on reddit because iirc you asked people not to respond to it in different places, doing that however makes the differences between communities of dubious value at best since there is going to be significant overlap between communities and you've taken steps to excise that overlap.
One, I specifically noted that as a complicating factor for the comparison in the original post. However, it shouldn't have any effect on the final summary of all 389 responses.

Two, I went and got responses from rpg.net and Reddit in an attempt to make it representative of a larger number of D&D players than I could get from ENWorld alone. Allowing folks to respond more than once to what was basically an extension of the original poll would have made the poll less representative.

You stated that "it doesn't sound like I could do a poll that would meet your standards." But what I am talking about are minimal standards of statistics as a science, not personal feelings, or some maximal high standard.
Well, yes, a statistically valid forum poll is highly improbable, maybe literally impossible.
If your standard is that I can't create a statistically valid poll using responses from D&D forums online, you're (at best) asking me to make this a research project on a scale that only a professional polling company would be able to achieve. Even the responses Wizards of the Coast get are from folks online who see links to their polls on internet forums, Facebook, Twitter, etc., so I'd be surprised if you consider those statistically valid...

If you or someone else is willing to put in such extraordinary amounts of effort to answer these questions, more power to you. I'll happily support it. But I only did this in the first place because it seemed like an interesting question to answer - I'm not interested in making a career out of this simply because the poll wasn't perfect.

Also, at this point, I'm pretty well convinced that you would dismiss any poll results I presented as "junk data", even if I did try to meet your professional-grade standards.


You know, it's kind of funny that these polls have been running for weeks now, but it's only after I present the final results that folks start attacking the validity of the poll itself...
 

Parmandur

Book-Friend
If your standard is that I can't create a statistically valid poll using responses from D&D forums online, you're (at best) asking me to make this a research project on a scale that only a professional polling company would be able to achieve.
But that's my point: short of doing that, you haven't actually learned anything.

WotC isn't trying to prove the validity of their results to me, but to internal stakeholders. We don't know what their full methodology and data entails, but they seem to be doing OK with it. If they wanted to publish their results for peer review (which is essentially what this thread is), then they would have to explain themselves further.

Also, at this point, I'm pretty well convinced that you would dismiss any poll results I presented as "junk data", even if I did try to meet your professional-grade standards.

No, if you could produce solid data meeting basic standards of scientific rigor, that would be fascinating. Honestly, it was vague curiosity about the final results that made me take a look after the fact.
 

JEB

Hero
But that's my point: short of doing that, you haven't actually learned anything.
I think I've learned a fair amount, at least about what folks on ENWorld. rpg.net, and various D&D subreddits want to see D&D keep in future editions. I would have liked to have more than 389 total responses (I'd been hoping for many more on Reddit, considering the size of their D&D user base) but unlike you, I don't think the opinions I got are meaningless. Imperfect, certainly, but not meaningless.

No, if you could produce solid data meeting basic standards of scientific rigor, that would be fascinating. Honestly, it was vague curiosity about the final results that made me take a look after the fact.
Since your "basic standards" would require me to conduct professional-grade polling well beyond my individual capabilities, I think you were doomed to disappointment...
 

Parmandur

Book-Friend
I think I've learned a fair amount, at least about what folks on ENWorld. rpg.net, and various D&D subreddits want to see D&D keep in future editions. I would have liked to have more than 389 total responses (I'd been hoping for many more on Reddit, considering the size of their D&D user base) but unlike you, I don't think the opinions I got are meaningless. Imperfect, certainly, but not meaningless.


Since your "basic standards" would require me to conduct professional-grade polling well beyond my individual capabilities, I think you were doomed to disappointment...
Every one if those couple responses has valid individual opinions...but, given the sampling methodology and the response rate, they say nothing about ENWorld users or Reddit posters, let alone D&D players at large. There is literally zero conclusions that can be reached from these responses.

The thing that gets me is, the task seems to require that level of large scale professional work to even be worth considering.
 

There's a certain misunderstanding of WotC intentions with UA polling there: UA is not for balance testing, but for market desirability testing. They can balance for power internally using their playtesting tools and math. UA is all about determining if people actually want an option at all. And they have started asking questions to sort out responses such as "What Edition did you start playing?", "Are you a DM?", "Were you interested in this sort of concept before you read this article, "Are you more likely to want to play this type after reading the article?"
Oh I agree, I deliberately picked an easy example to illustrate just for purposes of illustrating frustrations so it wouldn't need much context. Often wotc's polls do not even attempt to differentiate GM from player or ask questions relevant to both groups. As someone who is nearly always a gm & precovid for multiple tables a week (personal+AL games) I could only stare at the question in disbelief over how irrelevant they were & feel a bit of rage at the implication that GMs are apparently beneath consideration. They have indeed started getting better & I agree on that regard but it's a recent development & wotc has a long history of surveys that could be listed as figure1 in a section on how not to perform polls.


Questions attempting to gauge desirability from players & gms should generally by necessity be very different as their role responsibility & experience in any given session will be dramatically different. For example... is this a good addition to my table? Am I happy about this thing among my players? Am I confident in my ability to leverage this thing for interesting results? How well does this thing fit into my campaign world. so on & so forth. "Is this thing fun" is almost never ever relevant to me as a GM because my fun as a gm stems from juggling all the spinning plates & chainsaws collectively making up my game world & the campaign itself while doing things like working with my players to develop that tornado of buzzsaws & provide my players routes to accomplish goals in the process of being awesome together as an example.
 
Last edited:

One, I specifically noted that as a complicating factor for the comparison in the original post. However, it shouldn't have any effect on the final summary of all 389 responses.
It's not a "complicating factor", it's an injection of noise that ignores people active in multiple places are more likely to be more involved with d&d & possibly even spend more on it or any number of other things. What's worse is that 389 is likely a vanishingly tiny fraction of those communities even if you limit to those active within a short window of time like a week or month Those numbers are harder to gather for the various forums, but reddit shows number of members & number of online users for each subreddit on the right. /r/cheesemaking has 53.8k members & 41 online right now at 2am as a good example of a small niche subreddit. By comparison /r/dnd has 2.3m & 7.2k online right now at 2am est. It's a safe bet that 389 is less than the rounding error produced by rounding that 7.2k up or down.
Two, I went and got responses from rpg.net and Reddit in an attempt to make it representative of a larger number of D&D players than I could get from ENWorld alone. Allowing folks to respond more than once to what was basically an extension of the original poll would have made the poll less representative.
Yes and that's laudable for attempting, but with a single question & dubious sampling the biggest accomplishment of this poll is to provide ammunition for dismissing opinions with "posters on the forum are very different from the community at large" when dismissing it. That's not to say that good polling can't be done, statistics grads do it every so often as part of coursework & using all of 30seconds r so on google it looks like this might have been one.
If your standard is that I can't create a statistically valid poll using responses from D&D forums online, you're (at best) asking me to make this a research project on a scale that only a professional polling company would be able to achieve. Even the responses Wizards of the Coast get are from folks online who see links to their polls on internet forums, Facebook, Twitter, etc., so I'd be surprised if you consider those statistically valid...

If you or someone else is willing to put in such extraordinary amounts of effort to answer these questions, more power to you. I'll happily support it. But I only did this in the first place because it seemed like an interesting question to answer - I'm not interested in making a career out of this simply because the poll wasn't perfect.

Also, at this point, I'm pretty well convinced that you would dismiss any poll results I presented as "junk data", even if I did try to meet your professional-grade standards.


You know, it's kind of funny that these polls have been running for weeks now, but it's only after I present the final results that folks start attacking the validity of the poll itself...
People post forum polls all the time & it's generally not worth fighting over the poll itself since they don't say much unless it's a simple question like does x really happen with decisive results. Really all such a poll like the yoyo healing one at 77/26 with just over a hundred votes actually does though is lend a bit of weight to the discussion of the problem itself & any merits it might bring. There are tons of free & cheap polling tools that can be used online like surveymonkey & such that can be used in conjunction with a reasonably well worded survey to draw useful data from without even needing to know how to code your own survey application on an aws or something just like that one I dug up & linked to above did.

It's great that you found interest in doing some kind of poll, but people are offering you constructive advice not to attack you but to help provide you with the tools & knowledge needed to accomplish actually useful data collection if you decide that you enjoyed it enough to do better. They aren't offering this advice & knowledge because you ended the poll, it's being offered because you are presenting the results as something far more than they are.
 

JEB

Hero
It's not a "complicating factor", it's an injection of noise that ignores people active in multiple places are more likely to be more involved with d&d & possibly even spend more on it or any number of other things.
So... how exactly was I supposed to avoid that particular bit of "noise"? Seems like a damned if you do, damned if you don't situation.

What's worse is that 389 is likely a vanishingly tiny fraction of those communities even if you limit to those active within a short window of time like a week or month Those numbers are harder to gather for the various forums, but reddit shows number of members & number of online users for each subreddit on the right. /r/cheesemaking has 53.8k members & 41 online right now at 2am as a good example of a small niche subreddit. By comparison /r/dnd has 2.3m & 7.2k online right now at 2am est. It's a safe bet that 389 is less than the rounding error produced by rounding that 7.2k up or down.
Obviously, I would have really liked to get more than 389 responses, and I certainly expected to get more, since the D&D subreddits have such substantive user bases (I even posted in multiple ones, at the suggestion of folks here). But in practice, most of the initial responses came in the first few days, and then it dwindled to maybe a few a day, if that.

Unfortunately, I don't really see how there's a way for me to solve that problem. I can't force people to respond to a poll. Maybe if I mailed respondents cookies?

Yes and that's laudable for attempting, but with a single question & dubious sampling the biggest accomplishment of this poll is to provide ammunition for dismissing opinions with "posters on the forum are very different from the community at large" when dismissing it. That's not to say that good polling can't be done, statistics grads do it every so often as part of coursework & using all of 30seconds r so on google it looks like this might have been one.
Honestly, I'm astounded that they got nearly 2,500 responses, and in what sounds like relatively short order. I tried to see what they did differently, but if anything they seemed to do even less than I did, just a survey link. In any case, good for them! Certainly makes for a much stronger set of results than mine.

Of course, by certain standards, I suppose 2,500 doesn't mean much either...
 
Last edited:

Parmandur

Book-Friend
So... how exactly was I supposed to avoid that particular bit of "noise"? Seems like a damned if you do, damned if you don't situation.
That's why I wouldn't have bother d without some serious resources: you literally don't have any more information than you did before you started.
 


Level Up!

An Advertisement

Advertisement4

Top