D&D General What elements should D&D keep? forums vs. Reddit

[JEB]

If classes were a ‘nice to have’ and we’re a non-controversial option that didn’t really factor into the game. Then it might not be much more important option. They fact that you’ve picked a substantial element of the game clouds your point.

Saving throws and Deities is maybe a bit more difficult to weight importance though for instance. People said they would keep saving throws but not how important they were to keep.

Do you seriously think it's meaningless that stuff towards the top of the rankings is stuff like classes, hit points, levels, etc., while stuff in the lower end tends to be more specialized and controversial stuff like Challenge Ratings and alignment? What specific elements in the high end of the scale do you think would be examples of this phenomenon you suggest?

And supposing this phenomenon is in force (which I don't), this would only really apply to stuff in the higher end of the list, giving them an artificial bump. It might in fact suggest that the lower-ranking stuff is even less popular than my rankings suggest, because a large amount of people were indifferent to them and couldn't be bothered to vote in support of them.

Really don’t understand all the pushback on the feedback. I would think it’s a really uncontroversial point.

I'm pushing back because you're implying that the data in the poll is meaningless, that the opinions of 389 people don't actually reflect the relative value people place on those particular game elements. Which I think is rather silly.

If 90% of respondents want to keep classes, and 9% want to keep the World Axis, then in the grand scheme of things it's more important to a larger number of players to keep classes than the World Axis.

WotC has said in the past that forum opinion (which would include the equally self-selected relatively small part of the player base in the Reddit poster community) runs almost directly counter to actual broader player opinion.

Ah, so 91% want to keep the World Axis, and only 10% want to keep classes?

 

[JEB]

It could mean that people are fine with things like "Explorer's Pack" and "Scholar's Pack" rather than having to go shopping for each individual canteen, crowbar, and iron ration.

Yeah, that makes sense. Especially since 5E, the current standard-bearer of the game, encourages that through the default class builds.

Personally, I wasn't too sure what was meant by "lists of <whatever>," so I didn't check any of those. Like, I didn't know if that meant "this is a list of Sorcerer spells and this is a list of Bard spells" or something else entirely.

I'm disappointed to hear that. I thought that meaning was fairly clear, that it was itemized lists you could reference containing various specific pieces of equipment, magical spells, magic items, etc. But apparently not. Oh well, too late now...

 

[Parmandur]

I'm pushing back because you're implying that the data in the poll is meaningless, that the opinions of 389 people don't actually reflect the relative value people place on those particular game elements.

I mean, yeah, the "data" is meaningless noise. 389 self-selected respondents from a convenience sample do not meaningfully reflect 40 million people.

Ah, so 91% want to keep the World Axis, and only 10% want to keep classes?

That would be about as valid as any other conclusion drawn from these surveys, yes.

 

[Parmandur]

I'm just saying, there isn't even a margin of error at play here (I would it somewhere near +/- 100%, pure noise), due to sampling issues and survey design as @TheSword tried to explain in some constructive criticism.

This would not pass muster in peer review.

 

[Faolyn]

I'm disappointed to hear that. I thought that meaning was fairly clear, that it was itemized lists you could reference containing various specific pieces of equipment, magical spells, magic items, etc. But apparently not. Oh well, too late now...

Sorry, didn't get that at all.

 

[TheSword]

Do you seriously think it's meaningless that stuff towards the top of the rankings is stuff like classes, hit points, levels, etc., while stuff in the lower end tends to be more specialized and controversial stuff like Challenge Ratings and alignment? What specific elements in the high end of the scale do you think would be examples of this phenomenon you suggest?

And supposing this phenomenon is in force (which I don't), this would only really apply to stuff in the higher end of the list, giving them an artificial bump. It might in fact suggest that the lower-ranking stuff is even less popular than my rankings suggest, because a large amount of people were indifferent to them and couldn't be bothered to vote in support of them.


I'm pushing back because you're implying that the data in the poll is meaningless, that the opinions of 389 people don't actually reflect the relative value people place on those particular game elements. Which I think is rather silly.

At no point did I say the data was meaningless. It shows how widespread preferences are. It doesn’t show how important those preferences were.

I think it’s your use of the word importance, which in survey terms usually signifies strength of feeling.

If given the choice I would keep the wizards spell alarm in a new edition. I suspect many people would. It is in no way important to a 6th edition though, even if 100% of people ticked ‘would you keep the alarm spell in 6e. It is relevant to , but not important.

The danger is that you’re defending the veracity of your survey, by saying that it’s telling you what you expected to see. That’s called confirmation bias.

If at this point you don’t get what I’m saying, I’m probably wasting my time. It’s just worth people considering for future polls and surveys that want to examine importance or strength of feeling.

 

[JEB]

At no point did I say the data was meaningless. It shows how widespread preferences are. It doesn’t show how important those preferences were.

I think it’s your use of the word importance, which in survey terms usually signifies strength of feeling.

When someone says they want to keep an element of a game, are you suggesting that keeping that element in the game has no value to them? If it had no value to them, I would have expected them not to check the box that indicates they want to keep the element. That was, in fact, the intent of my instructions. And if they think it's valuable to keep... does that not suggest at least some degree of importance to them?

I suppose you can separate "I want to keep this element" from "I feel this element is important to keep", but that seems like splitting hairs to me.

If given the choice I would keep the wizards spell alarm in a new edition. I suspect many people would. It is in no way important to a 6th edition though, even if 100% of people ticked ‘would you keep the alarm spell in 6e. It is relevant to , but not important.

So if 100% of people said they wanted to keep alarm, you don't think that maybe Wizards might want to keep alarm in the game? That's certainly data I would like to have, if I was forced to choose between alarm and other spells, particularly ones that had very little explicit support.

That doesn't mean you absolutely can't cut alarm, it probably isn't a deal-breaker in the big picture... but why do so when so many folks say they want to keep it?

The danger is that you’re defending the veracity of your survey, by saying that it’s telling you what you expected to see. That’s called confirmation bias.

Actually, I was rather surprised by how high and low certain elements ranked. (I was certainly expecting alignment to do better on Reddit, for example, but it was actually pretty consistent with forum results. And it's interesting that an element as imprecise and arguably archaic as Armor Class has consistently been a strong contender. And I expected character races to do more poorly than they did, in the context of recent events and the "feel" poll.)

It is true, however, that on seeing the final tally, it's not that surprising to me that the highest-ranking elements are ones pretty integral to the historical structure of the game. Classes, hit points, ability scores, even the funky dice. But I didn't go in expecting those to do that well.

If at this point you don’t get what I’m saying, I’m probably wasting my time. It’s just worth people considering for future polls and surveys that want to examine importance or strength of feeling.

As I said, there's certainly room for polls that capture more nuance. I certainly would like to see someone do them.

I'll also say this - if anyone wants to have the raw data, to make their own analysis, I'm happy to share it. Google Forms actually gives you each individual response, anonymized, which could be very useful for figuring out bigger trends. (I don't see a way to get individual responses from ENWorld's poll, unfortunately, but maybe someone is listening and can help?)

 

[JEB]

Just to be clear, I'm not saying this poll is perfect, or that it couldn't have been improved, or that 389 results represent the feelings of every single D&D player on the planet. Those would be silly assertions; even the polls Wizards does themselves don't meet that standard, and they're certainly more comprehensive than my simple little survey.

But I do think the results have meaning for the respondents, and likely reflect the general preferences of the forums I polled. And they may possibly reflect some larger trends in the D&D community, since there were some broadly consistent patterns. Maybe the results are a complete fluke, but I don't think that's likely.

 

[Parmandur]

So if 100% of people said they wanted to keep alarm, you don't think that maybe Wizards might want to keep alarm in the game?

The standard WotC uses to make those sorts of decisions in their surveys are an average of 4/5 on a 5-point scale. If a UA Subclass gets a 3 from everyone, which would be a check on your survey, it doesn't make the cut. That's the point: these are matters that require a sliding scale, not a binary consideration.

 

[Parmandur]

But I do think the results have meaning for the respondents, and likely reflect the general preferences of the forums I polled. And they may possibly reflect some larger trends in the D&D community, since there were some broadly consistent patterns. Maybe the results are a complete fluke, but I don't think that's likely.

That's the thing, though, there is no way to demonstrate from the data any of those suppositions. I actually don't care about any of the actual points in the final results one way or another, more the principles of statistical analysis and survey design: I don't care for the World Axis, and my hypothesis is that 9% of players at large don't even know what it is. That nearly 10% of your respondents are holding a flame for the World Axis is one of the signs of a sampling issue.