D&D (2024) The WotC Playtest Surveys Have A Flaw

[Warpiglet-7]

Aren't you then selecting the participants rather than it being self-selecting though? I.e. you start with a representative sample, and then simply have to get their opinions.

sample if randomly selected would potentially be representative

FFS! I did not read all of the responses…seeing them now. This is probably redundant….

In a study, the problem can be a selection bias.

I am not saying this is true! But let’s pretend a bigger proportion of fans are in an age or cultural group that does not use computers much.

If you pull your sample via online/computer your sample would not be a representative one.

Again, I don’t think that is an issue itself here just using for illustration! But to your question, even a large sample could be a poor one if it is biased.

With appropriate sampling, you could have a much much smaller sample than this be a good representative one.

 

[UngainlyTitan]

And if those smaller ones are done right... I just usually don't have faith the extra effort is usually made.

Well yeah, you can always assume incompetence.

And they may not care about how well they're estimating the actual audience percent and just want to get lots of written feedback to make sure they haven't missed anything, get a summary value that gives a general feeling which way the wind is blowing, and have an easy to implement cut-off that sounds good to the masses.

That sounds expensive and of no particular value. If one is collecting a lot of data and spending money to do so, then spending an extra few bob to extract value from it is a no brainer.

 

[Cadence]

That sounds expensive and of no particular value. If one is collecting a lot of data and spending money to do so, then spending an extra few bob to extract value from it is a no brainer.

Does it really matter if the true percent that like the proposed change is 62% or 70% or 78% unless the 70% target wasn't just pulled from thin air as something that seemed plausible? I would guess the real goal is to make changes that aren't obviously really bad (like missing an obvious broken card in MtG in the inhouse play testing) and that don't seem to bring up a huge amount of hatred.

I'm curious what process you're proposing to check that wouldn't cost much that would give them that precision? But a I'm fine with it in a DM and would happily take an example from some other survey/industry.

 

[Ruin Explorer]

So what?
As in election,
Either you vote or you get to STFU and wait for results.

ROFL that's the one of most ridiculous things I've ever read.

No-one would respect an election that only 0.4 of the population voted on. Nor should they.

There is no self-selection bias present which would impact approval rating involved.

Absolute gibberish. The self-selection bias is likely huge, given the nature of the survey and that it even drives away people here.

Also self-selection is accounted for? Unsourced, unsupported rubbish. I work with people who do surveys for a living as well, and self-selection bias is absolutely an issue in our industry (law), so your unsourced, unsupported claim that it's a "solved problem" (a rather extreme claim) seems completely nonsensical.

And gamers? We know the self-selection with gamers means haters are much more likely to finish surveys than people who love something. But the exact percentage varies, so you can't just say "Solved!". Especially not when you're touting a 70% threshold for approval, rather than saying you use a complicated weighted formula. Or are we to presume they're lying re: the 70%?

The extreme bulk majority are ordinary DNDBeyond users.

Unless you have secret insider information, you have absolutely no basis to make that claim. It's just straightforwardly a lie presented as a fact. We cannot be certain who is answering these surveys, only that it's very few people (even compared to the users of D&D Beyond, which was 13 million last time WotC released figures, shortly after they bought it), and that surveys are extremely onerous and completely lack direction and focus.

I mean, unless all you mean by "ordinary DNDBeyond users" is "not people who post on ENworld, in which case it's a pointless and obvious truism. But it seems like you mean more than that.

At this point it looks very like you're just making up a bunch of "facts" that are convenient for you. You no provide no links, no sources, just claims.

 

[UngainlyTitan]

Does it really matter if the true percent like the proposed change is 62% or 70% or 78% unless the 70% target wasn't just pulled from thin air as something that seemed plausible? I would guess the real goal is to make changes that aren't obviously really bad (like missing an obvious broken card in MtG in the inhouse play testing) and that don't seem to bring up a huge amount of hatred.

I'm curious what process you're proposing to check that wouldn't cost much that would give them that precision? But a I'm fine with it in a DM and would happily take an example from some other survey/industry.

I think we may be arguing at cross purposes. I took your post, that I initially responded to as dismissing the surveys as having no statistical validity, my apologies if that is not the case. That, the WoTC statisticians are basically incompetent. It certainly appears to be the position some on this thread are taking.
My view is that a lot of money is being spent, they might as well do it right.
As for cross checking, you only need really small sample sizes to correlate a result. Add it in as part of their normal market research.

 

[Cadence]

I think we may be arguing at cross purposes. I took your post, that I initially responded to as dismissing the surveys as having no statistical validity, my apologies if that is not the case. That, the WoTC statisticians are basically incompetent. It certainly appears to be the position some on this thread are taking.
My view is that a lot of money is being spent, they might as well do it right.
As for cross checking, you only need really small sample sizes to correlate a result. Add it in as part of their normal market research.

I think you're right on us talking past each other.

I was assuming the WotC surveys might very well be getting what WotC wanted. I was having trouble with that being the usual intro stat simple random sample homework problem type thing - because that doesn't seem easy to do well and actually get both the lack of bias and small margins of error (see political polling or all the things raised in the stat books), and because I was having a hard time thinking it (really precise and accurate estimates of percents) was really the kind of thing WotC would particularly care about for building a new edition.

 

[mamba]

They can cross check if the UA surveys are good samples by running smaller more focused surveys to determine if that is the case.

and if they do not agree you do what?

 

[mamba]

Again, I don’t think that is an issue itself here just using for illustration! But to your question, even a large sample could be a poor one if it is biased.

With appropriate sampling, you could have a much much smaller sample than this be a good representative one.

agreed, that is what I was getting at

 

[Scott Christian]

I think we may be arguing at cross purposes. I took your post, that I initially responded to as dismissing the surveys as having no statistical validity, my apologies if that is not the case. That, the WoTC statisticians are basically incompetent. It certainly appears to be the position some on this thread are taking.

The middle ground is they may not be incompetent, but they may not be able to get the data they need. This happens all the time in school districts. They have some good statisticians, yet because of the community's lack of responses, they get many things wrong: projections, percentages for classes, etc.
So it may be that WotC pays these statisticians, and the stats people can't acquire the information they need. And then, not wanting to hand over nothing, they hand over something with a mistake. And I know they will give a confidence level, but even that has been wrong at times. Like, very wrong.

 

[UngainlyTitan]

and if they do not agree you do what?

Kinda depends on where you want to go. If they are being conservative then they increase the approval threshold until it makes the systematic errors irrelevant.