D&D (2024) The WotC Playtest Surveys Have A Flaw

[Parmandur]

Yeah I really don't understand why you wouldn't want to use free playtesting these days.

You get what you pay for.

UA material are first pass drafts, before they go into mathematical balancing. The first step is "do people like this idea?" and then they pass it to through their large private network and their spreadsheets to hash out the math. They have hundreds of tables in their actual playtest network.

 

[Njall]

Classes are plug-in options for the system, yes. You could re0lace them all without changing the rules per se (the actual rules change proposals are in the Rules Gallery).

But they aren't looking for that level of detail in feedback, it's a satisfaction survey: how much do you like this option on a 1-5 scale? DPS or charop analysis isn't what UA is collecting. It's a te.perature test to see if people like an idea, before they do the hard crunch playtest on the math.

It might surprise you, but "satisfaction", for a bunch of people, means "playing this class doesn't suck". Which is pretty dependant on balance.
All the changes they've proposed are about mechanics. Mechanics need to be evaluated. Depending on how they feel, certainly, but also on how impactful they are.
Concepts? Yeah, sure, in that case, "me likey" is fine. But that's not what they're changing.
The feedback they're asking for is pretty specific and pretty detailed. You aren't just rating stuff, you're also prompted to provide a reason (which is optional, but still...).

 

[Parmandur]

It might surprise you, but "satisfaction", for a bunch of people, means "playing this class doesn't suck". Which is pretty dependant on balance.
All the changes they've proposed are about mechanics. Mechanics need to be evaluated. Depending on how they feel, certainly, but also on how impactful they are.
Concepts? Yeah, sure, in that case, "me likey" is fine. But that's not what they're changing.
The feedback they're asking for is pretty specific and pretty detailed. You aren't just rating stuff, you're also prompted to provide a reason (which is optional, but still...).

Crawford has explicitly said in the past that most people aren't optimizing at all, which is reflected in the responses they get apparently. So, no, that's not a major factor because most people responding to UA aren't thinking like that at all.

They do read the qualitative feedback, but the feelings gut check is what they are primarily collecting before they move on the balancing test phase.

 

[Njall]

Crawford has explicitly said in the past that most people aren't optimizing at all, which is reflected in the responses they get apparently. So, no, that's not a major factor because most people responding to UA aren't thinking like that at all.

They do read the qualitative feedback, but the feelings gut check is what they are primarily collecting before they move on the balancing test phase.

And, again, even assuming that's what they're going for, providing a comprehensive package of the stuff they intended to change rather than small, disjointed chunks of the rules would've given them a clear picture of what was going to stick after a single round of playtest and survey, rather than several months.

 

[Parmandur]

And, again, even assuming that's what they're going for, providing a comprehensive package of the stuff they intended to change rather than small, disjointed chunks of the rules would've given them a clear picture of what was going to stick after a single round of playtest and survey, rather than several months.

Hardly: smaller and more bite sized was wise, to get targeted responses about the proposals, particular proposals that were direct opposites.

 

[Njall]

Hardly: smaller and more bite sized was wise, to get targeted responses about the proposals, particular proposals that were direct opposites.

Targeted responses about what? Stuff that plays well with the original PHB, but might not with the new, revamped classes?
Even eschewing maths, you play the complete package, not the single element.
"Old ranger + new feat/spell" might well "feel" and play way different from "New ranger + new feat".

Also, the stuff they're changing is mostly mechanical.
Whether people are doing charop level analysis or just playing it and seeing how it feels, it doesn't change a thing, they're still going to evaluate the mechanics, and that's something that's part of a complex, interlocking system.

If the kind of feedback they're looking for is "yes, but how does this feat feel, taken in isolation?" instead, I'd argue that kind of feedback is even more useless, as they're getting an opinion that might not work well at the table, provided by a minuscule percentage of the player base, which might well not represent the larger picture.
That's not a playtest, that's a cop out.

 

[Parmandur]

Targeted responses about what? Stuff that plays well with the original PHB, but might not with the new, revamped classes?

Yes, same difference: remember, this is a backwards compatible revision, so everything needs to work nicely together, anyways. They are expecting and designing for people to mix and match freely.

Even eschewing maths, you play the complete package, not the single element.
"Old ranger + new feat/spell" might well "feel" and play way different from "New ranger + new feat".

Also, the stuff they're changing is mostly mechanical.
Whether people are doing charop level analysis or just playing it and seeing how it feels, it doesn't change a thing, they're still going to evaluate the mechanics, and that's something that's part of a complex, interlocking system.

Modern D&D is complex, but it is preciaylnot an interlocking system: it's loose, modular system that can plug options in or out.

If the kind of feedback they're looking for is "yes, but how does this feat feel, taken in isolation?"

Yes, that's all they are looking for.

instead, I'd argue that kind of feedback is even more useless, as they're getting an opinion that might not work well at the table, provided by a minuscule percentage of the player base, which might well not represent the larger picture.
That's not a playtest, that's a cop out.

As established upthread, the sampling size is excellent.

 

[Cadence]

For an opt in survey, you need a sample size of about 0.038% to get within 5% margin of error, WotC sample size is about 4x bigger than that. So with a 70% threshold, WotC should have high confidence that a minimum of 65% of the player base has approved of something. That put's it in 2/3rds consensus for change.

Do you have a good link explaining this? I'm wondering what assumptions on the opt-in mechanism are needed and/or which parameter [pct for which population] are they within 5% for. -- I'm kind of thinking of the Literary Digest poll of 1936.

For most of the margin of error calculations for random samples, doesn't population size [Edit: or ratio of sample to population size] only come in when accounting for the error between the actual finite (like hypergeomtric) calculation and the commonly used infinite one (like binomial) - unless maybe something about the sampling mechanism depends on the the population size? At the other extreme, you could end up a guaranteed 10% off even if you had 90% of the population if responding is most negatively associated with one of the views, right?

Yep. The trouble is, you need a sufficient sample size to determine how "representative" it really is, and that's all but impossible.

And even then (except for a random sample) is there a real way to show from the one sample ho representative it is in terms of the variables they actually care about? If the wizard lovers from demographic combination X are motivated to reply and the others from X aren't, then it doesn't matter how close they got to matchng demographic X overall, right? -- Thinking of the 1948 quota sampling picking Dewey.

 

[bert1001 fka bert1000]

You get what you pay for.

UA material are first pass drafts, before they go into mathematical balancing. The first step is "do people like this idea?" and then they pass it to through their large private network and their spreadsheets to hash out the math. They have hundreds of tables in their actual playtest network.

That's good to know.

That said, 5e v1 has some dubious stuff that the "actual playtest network" didn't catch that fans pointed out pretty quickly after release.

Of course, maybe they did do all the math and comparisons and it's a deliberate design choice. I never get a really good insight into true design goals.

 

[Parmandur]

That's good to know.

That said, 5e v1 has some dubious stuff that the "actual playtest network" didn't catch that fans pointed out pretty quickly after release.

Of course, maybe they did do all the math and comparisons and it's a deliberate design choice. I never get a really good insight into true design goals.

Well, some like the Four Elements Monk or Beastmaster works fine by the math, if played as intended, but is also frustrating to people. Which is why they are changing. Bit the math was right, it was the feels that were off.