D&D General What elements should D&D keep? forums vs. Reddit

[JEB]

Even the sampling is badly flawed, I answered the poll here & then ignored it when I saw it on reddit because iirc you asked people not to respond to it in different places, doing that however makes the differences between communities of dubious value at best since there is going to be significant overlap between communities and you've taken steps to excise that overlap.

One, I specifically noted that as a complicating factor for the comparison in the original post. However, it shouldn't have any effect on the final summary of all 389 responses.

Two, I went and got responses from rpg.net and Reddit in an attempt to make it representative of a larger number of D&D players than I could get from ENWorld alone. Allowing folks to respond more than once to what was basically an extension of the original poll would have made the poll less representative.

You stated that "it doesn't sound like I could do a poll that would meet your standards." But what I am talking about are minimal standards of statistics as a science, not personal feelings, or some maximal high standard.

Well, yes, a statistically valid forum poll is highly improbable, maybe literally impossible.

If your standard is that I can't create a statistically valid poll using responses from D&D forums online, you're (at best) asking me to make this a research project on a scale that only a professional polling company would be able to achieve. Even the responses Wizards of the Coast get are from folks online who see links to their polls on internet forums, Facebook, Twitter, etc., so I'd be surprised if you consider those statistically valid...

If you or someone else is willing to put in such extraordinary amounts of effort to answer these questions, more power to you. I'll happily support it. But I only did this in the first place because it seemed like an interesting question to answer - I'm not interested in making a career out of this simply because the poll wasn't perfect.

Also, at this point, I'm pretty well convinced that you would dismiss any poll results I presented as "junk data", even if I did try to meet your professional-grade standards.


You know, it's kind of funny that these polls have been running for weeks now, but it's only after I present the final results that folks start attacking the validity of the poll itself...

 

[Parmandur]

If your standard is that I can't create a statistically valid poll using responses from D&D forums online, you're (at best) asking me to make this a research project on a scale that only a professional polling company would be able to achieve.

But that's my point: short of doing that, you haven't actually learned anything.

WotC isn't trying to prove the validity of their results to me, but to internal stakeholders. We don't know what their full methodology and data entails, but they seem to be doing OK with it. If they wanted to publish their results for peer review (which is essentially what this thread is), then they would have to explain themselves further.

Also, at this point, I'm pretty well convinced that you would dismiss any poll results I presented as "junk data", even if I did try to meet your professional-grade standards.

No, if you could produce solid data meeting basic standards of scientific rigor, that would be fascinating. Honestly, it was vague curiosity about the final results that made me take a look after the fact.

 

[JEB]

But that's my point: short of doing that, you haven't actually learned anything.

I think I've learned a fair amount, at least about what folks on ENWorld. rpg.net, and various D&D subreddits want to see D&D keep in future editions. I would have liked to have more than 389 total responses (I'd been hoping for many more on Reddit, considering the size of their D&D user base) but unlike you, I don't think the opinions I got are meaningless. Imperfect, certainly, but not meaningless.

No, if you could produce solid data meeting basic standards of scientific rigor, that would be fascinating. Honestly, it was vague curiosity about the final results that made me take a look after the fact.

Since your "basic standards" would require me to conduct professional-grade polling well beyond my individual capabilities, I think you were doomed to disappointment...

 

[Parmandur]

I think I've learned a fair amount, at least about what folks on ENWorld. rpg.net, and various D&D subreddits want to see D&D keep in future editions. I would have liked to have more than 389 total responses (I'd been hoping for many more on Reddit, considering the size of their D&D user base) but unlike you, I don't think the opinions I got are meaningless. Imperfect, certainly, but not meaningless.


Since your "basic standards" would require me to conduct professional-grade polling well beyond my individual capabilities, I think you were doomed to disappointment...

Every one if those couple responses has valid individual opinions...but, given the sampling methodology and the response rate, they say nothing about ENWorld users or Reddit posters, let alone D&D players at large. There is literally zero conclusions that can be reached from these responses.

The thing that gets me is, the task seems to require that level of large scale professional work to even be worth considering.

 

[tetrasodium]

There's a certain misunderstanding of WotC intentions with UA polling there: UA is not for balance testing, but for market desirability testing. They can balance for power internally using their playtesting tools and math. UA is all about determining if people actually want an option at all. And they have started asking questions to sort out responses such as "What Edition did you start playing?", "Are you a DM?", "Were you interested in this sort of concept before you read this article, "Are you more likely to want to play this type after reading the article?"

Oh I agree, I deliberately picked an easy example to illustrate just for purposes of illustrating frustrations so it wouldn't need much context. Often wotc's polls do not even attempt to differentiate GM from player or ask questions relevant to both groups. As someone who is nearly always a gm & precovid for multiple tables a week (personal+AL games) I could only stare at the question in disbelief over how irrelevant they were & feel a bit of rage at the implication that GMs are apparently beneath consideration. They have indeed started getting better & I agree on that regard but it's a recent development & wotc has a long history of surveys that could be listed as figure1 in a section on how not to perform polls.


Questions attempting to gauge desirability from players & gms should generally by necessity be very different as their role responsibility & experience in any given session will be dramatically different. For example... is this a good addition to my table? Am I happy about this thing among my players? Am I confident in my ability to leverage this thing for interesting results? How well does this thing fit into my campaign world. so on & so forth. "Is this thing fun" is almost never ever relevant to me as a GM because my fun as a gm stems from juggling all the spinning plates & chainsaws collectively making up my game world & the campaign itself while doing things like working with my players to develop that tornado of buzzsaws & provide my players routes to accomplish goals in the process of being awesome together as an example.

 

[tetrasodium]

One, I specifically noted that as a complicating factor for the comparison in the original post. However, it shouldn't have any effect on the final summary of all 389 responses.

It's not a "complicating factor", it's an injection of noise that ignores people active in multiple places are more likely to be more involved with d&d & possibly even spend more on it or any number of other things. What's worse is that 389 is likely a vanishingly tiny fraction of those communities even if you limit to those active within a short window of time like a week or month Those numbers are harder to gather for the various forums, but reddit shows number of members & number of online users for each subreddit on the right. /r/cheesemaking has 53.8k members & 41 online right now at 2am as a good example of a small niche subreddit. By comparison /r/dnd has 2.3m & 7.2k online right now at 2am est. It's a safe bet that 389 is less than the rounding error produced by rounding that 7.2k up or down.

Two, I went and got responses from rpg.net and Reddit in an attempt to make it representative of a larger number of D&D players than I could get from ENWorld alone. Allowing folks to respond more than once to what was basically an extension of the original poll would have made the poll less representative.

Yes and that's laudable for attempting, but with a single question & dubious sampling the biggest accomplishment of this poll is to provide ammunition for dismissing opinions with "posters on the forum are very different from the community at large" when dismissing it. That's not to say that good polling can't be done, statistics grads do it every so often as part of coursework & using all of 30seconds r so on google it looks like this might have been one.

If your standard is that I can't create a statistically valid poll using responses from D&D forums online, you're (at best) asking me to make this a research project on a scale that only a professional polling company would be able to achieve. Even the responses Wizards of the Coast get are from folks online who see links to their polls on internet forums, Facebook, Twitter, etc., so I'd be surprised if you consider those statistically valid...

If you or someone else is willing to put in such extraordinary amounts of effort to answer these questions, more power to you. I'll happily support it. But I only did this in the first place because it seemed like an interesting question to answer - I'm not interested in making a career out of this simply because the poll wasn't perfect.

Also, at this point, I'm pretty well convinced that you would dismiss any poll results I presented as "junk data", even if I did try to meet your professional-grade standards.


You know, it's kind of funny that these polls have been running for weeks now, but it's only after I present the final results that folks start attacking the validity of the poll itself...

People post forum polls all the time & it's generally not worth fighting over the poll itself since they don't say much unless it's a simple question like does x really happen with decisive results. Really all such a poll like the yoyo healing one at 77/26 with just over a hundred votes actually does though is lend a bit of weight to the discussion of the problem itself & any merits it might bring. There are tons of free & cheap polling tools that can be used online like surveymonkey & such that can be used in conjunction with a reasonably well worded survey to draw useful data from without even needing to know how to code your own survey application on an aws or something just like that one I dug up & linked to above did.

It's great that you found interest in doing some kind of poll, but people are offering you constructive advice not to attack you but to help provide you with the tools & knowledge needed to accomplish actually useful data collection if you decide that you enjoyed it enough to do better. They aren't offering this advice & knowledge because you ended the poll, it's being offered because you are presenting the results as something far more than they are.

 

[JEB]

It's not a "complicating factor", it's an injection of noise that ignores people active in multiple places are more likely to be more involved with d&d & possibly even spend more on it or any number of other things.

So... how exactly was I supposed to avoid that particular bit of "noise"? Seems like a damned if you do, damned if you don't situation.

What's worse is that 389 is likely a vanishingly tiny fraction of those communities even if you limit to those active within a short window of time like a week or month Those numbers are harder to gather for the various forums, but reddit shows number of members & number of online users for each subreddit on the right. /r/cheesemaking has 53.8k members & 41 online right now at 2am as a good example of a small niche subreddit. By comparison /r/dnd has 2.3m & 7.2k online right now at 2am est. It's a safe bet that 389 is less than the rounding error produced by rounding that 7.2k up or down.

Obviously, I would have really liked to get more than 389 responses, and I certainly expected to get more, since the D&D subreddits have such substantive user bases (I even posted in multiple ones, at the suggestion of folks here). But in practice, most of the initial responses came in the first few days, and then it dwindled to maybe a few a day, if that.

Unfortunately, I don't really see how there's a way for me to solve that problem. I can't force people to respond to a poll. Maybe if I mailed respondents cookies?

Yes and that's laudable for attempting, but with a single question & dubious sampling the biggest accomplishment of this poll is to provide ammunition for dismissing opinions with "posters on the forum are very different from the community at large" when dismissing it. That's not to say that good polling can't be done, statistics grads do it every so often as part of coursework & using all of 30seconds r so on google it looks like this might have been one.

Honestly, I'm astounded that they got nearly 2,500 responses, and in what sounds like relatively short order. I tried to see what they did differently, but if anything they seemed to do even less than I did, just a survey link. In any case, good for them! Certainly makes for a much stronger set of results than mine.

Of course, by certain standards, I suppose 2,500 doesn't mean much either...

 

[Parmandur]

So... how exactly was I supposed to avoid that particular bit of "noise"? Seems like a damned if you do, damned if you don't situation.

That's why I wouldn't have bother d without some serious resources: you literally don't have any more information than you did before you started.

 

[JEB]

Comparing these results to "What makes D&D feel like D&D?": D&D General - D&D's feel vs. what D&D should keep - final comparison