D&D (2024) The WotC Playtest Surveys Have A Flaw

[Stalker0]

Some of these comments have a real, “statistics, how do they work?” kind of vibe going on.

Yeah having caught up on the thread, what this is telling me is that a lot of people just don't understand basic sampling statistics.

The entire point of statistical sample theory is that once you capture a certain number of people, you can confidently use those results to apply to the greater whole, even if that whole is much much much MUCH larger than the sample. The numbers that WOTC has are not only a good sample, they are an AMAZING one. Like this is the kind of survey results all pollsters dream of.

Now...the trick is, polling a representative sample of your audience, and that's a question mark. Obviously wotc is going to have a bias in online people who take their survey, or whatever demographics visit their site. Those are potential issues you could poke out the survey and say "this might be a problem".

But the actual number of surveys, oh no that is a wonderful number, way more than WOTC needs to confidentially project results.

 

[Parmandur]

Yeah having caught up on the thread, what this is telling me is that a lot of people just don't understand basic sampling statistics.

The entire point of statistical sample theory is that once you capture a certain number of people, you can confidently use those results to apply to the greater whole. The numbers that WOTC has are not only a good sample, they are an AMAZING one. Like this is the kind of survey results all pollsters dream of.

Now...the trick is, polling a representative sample of your audience, and that's a question mark. Obviously wotc is going to have a bias in online people who take their survey, or whatever demographics visit their site. Those are potential issues you could poke out the survey and say "this might be a problem".

But the actual number of surveys, oh no that is a wonderful number, way more than WOTC needs to confidentially project results.

Hopefully they use the demographic details they ask to help with analyzing the sample data...

 

[Thomas Shey]

Yeah having caught up on the thread, what this is telling me is that a lot of people just don't understand basic sampling statistics.

The entire point of statistical sample theory is that once you capture a certain number of people, you can confidently use those results to apply to the greater whole, even if that whole is much much much MUCH larger than the sample. The numbers that WOTC has are not only a good sample, they are an AMAZING one. Like this is the kind of survey results all pollsters dream of.

Now...the trick is, polling a representative sample of your audience, and that's a question mark. Obviously wotc is going to have a bias in online people who take their survey, or whatever demographics visit their site. Those are potential issues you could poke out the survey and say "this might be a problem".

But the actual number of surveys, oh no that is a wonderful number, way more than WOTC needs to confidentially project results.

My reaction on reading this from the start (and to be clear, I don't really have a dog in the fight--as I've noted before I'm not a D&D person proper) is I'm more dubious about the methodology than the numbers. It sounds like they may well be, effectively, screening out some non-trivial parts of their audience they don't want to be if they want it to serve a purpose other than being able to say they did surveys.

 

[MichaelSomething]

I though us Enworlders would be happy that we have the most power to influence the game to fit our desires?

I don't want another 4E where WOTC went outside of our sphere to pick up a bunch of unpopular ideas!

 

[payn]

I though us Enworlders would be happy that we have the most power to influence the game to fit our desires?

I don't want another 4E where WOTC went outside of our sphere to pick up a bunch of unpopular ideas!

Am I on the wrong EN world? Pretty sure the one I'm used to would survey to end 5E and reprint 4E.

 

[Zardnaar]

Am I on the wrong EN world? Pretty sure the one I'm used to would survey to end 5E and reprint 4E.

And get the same reception.

 

[DammitVictor]

It kinda hurts to step away from the brand and fandom. There's a lot of history and memories there.

It gets better. The D&D you loved is still as good as it ever was, and in the 21st century... it's going to continue getting better every year as third-party publishers continue supporting it. You will even see third-party conversions of all the new stuff you're missing out on, making all the old content of older editions of D&D and all the new content of newer editions of D&D available for the version you're playing.

No matter what Hasbro tries to do with the OGL in the future, it will continue doing its job-- nobody will ever be able to take D&D away from us.

 

[Retreater]

Juat as an aside, the evidence suggests that DMs do almost all the book buying: there aren't any 5E products that aren't essentially DM products.

Yet all the playtests and surveys are focused on player options. What do players want to sell more books? What new powers would they like for their characters? Do they think this new ancestry is cool?

How about questions like...

• Did you like these new monster abilities?
• Was this stat block format an improvement?
• Did this new encounter math let your party survive one encounter? How about four encounters?
• When you tried running your 5th level party with this assortment of magical gear, how did the challenge feel compared to no magical gear?
• Have you tried flanking in your game? How did that work?
• What about disarming maneuvers? Are those effective?
• Do you ever use gritty rest variants?

In general, they aren't asking questions that matter.

I'm not using Beyond tools, but it seems most of what you are asking for is available there...?

I don't think so. There's no PDF access. If there are tools, they're all subscription-based. If there's anything searchable, it's for finding the content on Beyond, and not really useful for in-person gaming with actual books.

And theybare putting out a revised DMG shortly, which looks to address precisely many of these concerns.

In a year. It should've been revised far sooner than that. Or given us a DMG2. Or revised Encounter Guidelines. Or anything.

And again, I fail to see how anything that actively made DMs happy wouldn't sell better? One big issue might be that different people need different things from the game, amd WotC is aiming for the mass D&D market.

Conspiracy theory time. They don't want DMs to create content. They want us purchasing their IP - because that's where the money is. If we're given functional encounter guidelines, good monster stat blocks, we're going to be less likely to create our own worlds and adventures. Keeping us in the ecosystem is important.
I'm not sure they even want us playing the game. Purchasing books, watching the movies, buying stuffed animals - yes, all that stuff. But I don't think they care about the part of the hobby that brings us to ENWorld. They have said that the game is only one part of their plan for the brand - and it's a very costly, time-consuming part of it that doesn't pay off as well as (for example) the D&D Cookbook, or partnering with Larian to make Baldur's Gate 3.

 

[doctorbadwolf]

This thread is forked from another, unrelated thread. Things started to drift off-topic there, but I felt it's a good topic for discussion.


This is a big one.

I belong to three different gaming groups. And of those 18 people in total, I'm the only one who is following the development of the game at all. I'm the only one who knew that Wizards of the Coast is working on a new edition rules revision, and my fellow gamers get really defensive when I mention it. One guy will actually growl at me every time I bring up the playtest, "We are not changing editions again!" They weren't even aware of the OGL debacle earlier this year, and it was supposed to have blown up the Internet.

These are people that I game with every week. We schedule and coordinate our games through social media, so it's not like they live under a rock either. But as one person out of eighteen total, I'm literally the only 6% of active 5E gamers that I know of who are even aware of these playtests. I imagine the number of folks who are aware and interested is even lower. How much less, then, for the number of people who are (a) interested enough to (b) download the material, (c) read it, (d) playtest it, and (e) provide feedback?

And that fraction of a fraction of a fraction of people that made it all the way to Step (e) is supposed to be everyone's voice in the room.

I don't have a better idea, but still. That's a big ask.

I mean, small but representative samples is generally the best ya get.

 

[CleverNickName]

I mean, small but representative samples is generally the best ya get.

Yep. The trouble is, you need a sufficient sample size to determine how "representative" it really is, and that's all but impossible.

This leads me to believe that they aren't relying completely on these playtest surveys. Probably not even mostly. I'd wager they give far more weight to their own internal playtesters and designers, and just watch the survey results for trends and hot topics. They pretty much have to. (But that's just a friendly wager; I'm not putting money on it. I have no idea how that particular sausage is getting made, and I'm pretty sure I don't even want to know.)