D&D (2024) The WotC Playtest Surveys Have A Flaw

[mamba]

Scientifically, you can get "representative samples" for surveys or data collection and analysis with far fewer participants.

Aren't you then selecting the participants rather than it being self-selecting though? I.e. you start with a representative sample, and then simply have to get their opinions.

 

[Cadence]

Aren't you then selecting the participants rather than it being self-selecting though? I.e. you start with a representative sample, and then simply have to get their opinions.

If you have a simple random sample (all groups of size n have the same chance of being picked) from your target population and they all responded honestly, then it is an unbiased sample for every single variable. That is, on average it will work out right if you imagine repeatedly doing the sampling, but your particular sample could be off. How far off you think it might be is described by the margin of error.

If you know the variable of interest is related to some demographic variables (sex, age, region, etc...) and know how the population breaks down in terms of those demographics, then a stratified random sample where you draw separately from each demographic will have smaller margins of error.

They don't have any of those good sampling things here, and they almost certainly don't have the information to make high quality adjustments like the best places do for political polls.

I kind of wonder if picking some varied focus groups and doing a qualitative analysis would be more useful. (I'm guessing/hoping they have something like that going on too?)

 

[UngainlyTitan]

Not really. Scientifically, you can get "representative samples" for surveys or data collection and analysis with far fewer participants. Now, I'm not a scientist, or a pollster, but I'm sure there are those on these boards who could explain it. Basically, the numbers that WOTC pulls for their surveys - other companies would kill to get that kind of response rate. It shouldn't be dismissed.

+edit: I might disagree with the results, but that doesn't make them non-representative. It just means I disagree with the direction the game is taking. And that has happened to me, and I've left 5e as a result.

They have the means to acquire extra validation data if they want. We do not have full information on what WoTC knows or how data is used. I get the impression that many see the surveys s a simple pass/fail type exercise but I get the impression that it is a more A/B type testing.
The accusation that the surveys are wrong, strongly implies that WoTC cannot employ professionals in the field. OK, it's possible but seems unlikely to me.
For that matter we never get the full results, or what these results are being compared to or what other data is informing decisions.

 

[Parmandur]

If you have a simple random sample (all groups of size n have the same chance of being picked) from your target population and they all responded honestly, then it is an unbiased sample for every single variable. That is, on average it will work out right if you imagine repeatedly doing the sampling, but your particular sample could be off. How far off you think it might be is described by the margin of error.

If you know the variable of interest is related to some demographic variables (sex, age, region, etc...) and know how the population breaks down in terms of those demographics, then a stratified random sample where you draw separately from each demographic will have smaller margins of error.

They don't have any of those good sampling things here, and they almost certainly don't have the information to make high quality adjustments like the best places do for political polls.

I kind of wonder if picking some varied focus groups and doing a qualitative analysis would be more useful. (I'm guessing/hoping they have something like that going on too?)

Crawford said explicitly in the last video thst they do focus tests, yes. And these survey responses not only capture demographic info...they are also tied to Beyond Accounts now. So no doubt they are able to duscern a number of things they aren't going to share thst they know.

 

[jasper]

so 3 out of 6 stopped playing because of proposed changes, is that what you are saying?

Why would they drop out now?

Yes. They did not drop out all at once. They knew that the changes were going to go through in one form or the other.

 

[Parmandur]

They have the means to acquire extra validation data if they want. We do not have full information on what WoTC knows or how data is used. I get the impression that many see the surveys s a simple pass/fail type exercise but I get the impression that it is a more A/B type testing.
The accusation that the surveys are wrong, strongly implies that WoTC cannot employ professionals in the field. OK, it's possible but seems unlikely to me.
For that matter we never get the full results, or what these results are being compared to or what other data is informing decisions.

WotC has specified thst UA is A-B testing, yup. In this case, A is the 2014 rules and the big surveys they did on them.

 

[Micah Sweet]

Yes. They did not drop out all at once. They knew that the changes were going to go through in one form or the other.

Was there a reason your group couldn't simply...not institute the changes at your table? The existence of a new book doesn't demand everyone "upgrade" to it.

 

[Horwath]

30m it's WotC's own estimate of how many people are playing D&D nowadays (I think late 2021?).

40K is the largest number WotC themselves have reported for playtest responses for 1D&D/2024.

IIRC, they noted the numbers were lower for a later 1D&D/2024 UA (20K, I think), but the 40K is definite.

70% is WotC's own figure, as a percentage of people who answer a UA, who must approve of any change.

So these are being multiplied together. 40k is 0.13% of 30m. 70% of 0.13 is 0.091%. That is the percentage of players WotC is relying on to approve stuff.


The numbers are based directly on WotC's own statements.

That is the best information we have available.

EDIT - Just to add to this, we could just assume WotC is delusional in their estimates and only 10m people are actually playing D&D, but it barely makes a difference. 40k just becomes 0.4%, and the "approval" % is then 0.28%, which whilst obviously larger than 0.091%, is still laughable in the same way.

So what?
As in election,
Either you vote or you get to STFU and wait for results.

 

[mamba]

They don't have any of those good sampling things here, and they almost certainly don't have the information to make high quality adjustments like the best places do for political polls.

that was my point

I kind of wonder if picking some varied focus groups and doing a qualitative analysis would be more useful. (I'm guessing/hoping they have something like that going on too?)

No idea, if they could get more detailed answers on DDB and their VTT (once that is around), they probably would get a much better idea than their surveys will ever give them. Not sure what they can actually track that way today however.

 

[Mistwell]

Again I ran the numbers in a professional survey calculator and it said these kinds of numbers not only had a high level of confidence at 95% but remained high level confidence at 99%.

And you don't deduct those who disagree - you don't deduct 30% because they only require 70%. That's not how statistics works either. This level of responses given this field of consumers has a 99% confidence level that these results accurately reflect the consumer population for these products. IE that 70%+ of the consumers of these products like the results of this playtest MORE than they liked the 2014 version (some of which was now scoring in the 20% region).