D&D (2024) The WotC Playtest Surveys Have A Flaw

[CleverNickName]

This thread is forked from another, unrelated thread. Things started to drift off-topic there, but I felt it's a good topic for discussion.

2) The surveys are being answered by about 0.01% of WotC's estimated current playerbase. That is not a good sample size. You have to jump through hoops and sign up to even answer them, that they are then incredibly tediously long, that their questions are not at all focused or thoughtful or specific, but utterly generalized and generic, and soon. This means that I suspect that 0.01% is largely the same sort of people - mostly grogs like myself - who answered the DND Next surveys a decade ago.

This is a big one.

I belong to three different gaming groups. And of those 18 people in total, I'm the only one who is following the development of the game at all. I'm the only one who knew that Wizards of the Coast is working on a new edition rules revision, and my fellow gamers get really defensive when I mention it. One guy will actually growl at me every time I bring up the playtest, "We are not changing editions again!" They weren't even aware of the OGL debacle earlier this year, and it was supposed to have blown up the Internet.

These are people that I game with every week. We schedule and coordinate our games through social media, so it's not like they live under a rock either. But as one person out of eighteen total, I'm literally the only 6% of active 5E gamers that I know of who are even aware of these playtests. I imagine the number of folks who are aware and interested is even lower. How much less, then, for the number of people who are (a) interested enough to (b) download the material, (c) read it, (d) playtest it, and (e) provide feedback?

And that fraction of a fraction of a fraction of people that made it all the way to Step (e) is supposed to be everyone's voice in the room.

I don't have a better idea, but still. That's a big ask.

 

[pointofyou]

Seems as though WotC's thinking is that the people answering the surveys know enough and care enough to do so. I am not convinced it's testing for the kind of knowledge or caring that leads to helpful input.

 

[Retreater]

Of my group - and their other groups - consisting of probably 30+ people, no one is taking the surveys because we have no "buy in." Wizards has said that they're not making any significant changes. We would only like D&D with significant changes. Ergo, we will not playtest or be interested in the developments.

 

[Oligopsony]

WotC doesn’t need to publicize how they do focus groups in the same way they need to publicize the survey, so I wouldn’t make absence of evidence into a strong evidence of absence. But yeah they should absolutely go around to random high school chess, theatre, and anime clubs, demo the game, take close notes and feedback, etc.

Surveys are easy to weight on demographics if you know the base demographics, and there’s probably some proxies in there. The qualitative responses will likely contain insights unrelated to that. But also this means that quoting raw approval numbers isn’t very illuminating.

 

[CleverNickName]

Of my group - and their other groups - consisting of probably 30+ people, no one is taking the surveys because we have no "buy in." Wizards has said that they're not making any significant changes. We would only like D&D with significant changes. Ergo, we will not playtest or be interested in the developments.

I have the opposite problem. I've been told, repeatedly, that we aren't changing editions "ever again." Even after a decade, my friends are still salty over the 3.5E/Pathfinder/4E wreck. We're going to play 5th Edition forever, with or without WotC.

 

[Ruin Explorer]

Sorry, responded in the other thread but yeah my experience is broadly similar. The other three D&D DMs I play with are all good DMs, all have "opinions" about 5E, both positive and negative, and a lot of the players do too, but absolutely none of them have engaged with the playtest, even though all of them are on D&D Beyond and have accounts. The most I got was convincing one DM to download one packet and try the survey, but he gave up on it.

These are probably more serious than average D&D players and DMs, and they're older and more likely to have the time and the nerdy interest to fill in those surveys - but they're still not doing it!

I don't blame them, I blame the process, and WotC for relying on that process.

Now, that might not be Crawford's fault, let's be clear - whilst he seems like the type to just stick to an existing process, that's me being judgemental. What I suspect is that case is that WotC simply has not provided resources for doing research into what the playerbase wants in a more proactive way that reaches a larger proportion of the playerbase.

And I think Crawford and his team are struggling even with the surveys. I work with surveys some of the time professionally, and certainly with the art of asking questions, and it is very much an art, constructing a good survey - I'm sure there's scientific approaches as well - but you've got to know what you want to get out of the survey, exactly why you're conducting it, and why you're asking questions.

I've seen the statistics from stuff I've worked with where one survey gets huge engagement in terms of people opening up, but hardly anyone finishes it, and I've seen others where virtually everyone who opened it, whether few or many, finished it. And the latter? They tend to be shorter, more focused, to ask real questions which need to asked, not to just rate thing after thing after thing. And talking to the people trying to derive information from them, the more focused surveys, with questions specifically there to determine where the issues are have proven much more useful than the rate this rate this rate this rate this surveys. The latter aren't without value, but they tend to be more box-checking exercises rather than things that really influence how the company operates - so it's interesting that WotC is basing so much decision-making on this.

Of my group - and their other groups - consisting of probably 30+ people, no one is taking the surveys because we have no "buy in." Wizards has said that they're not making any significant changes. We would only like D&D with significant changes. Ergo, we will not playtest or be interested in the developments.

That's interesting and I think definitely part of why people don't care. WotC have sort of tried to have their cake and eat it, but saying they're doing this new edition, and also they're not changing anything, don't be afraid, and what it seems like an awful lot of people have heard is "You have no reason to care". And this means no engagement.

And I wonder what that will means for sales of 2024. It's a bit early to doom-monger or whatever, but the fact that nobody seems to care but us very online types, to me, that doesn't bode well for the sales of 2024.

Whereas a new edition which might mean some real changes or improvements would probably have annoyed plenty of people (including @CleverNickName's people who were saying "Not another bloody edition!"), but at least people would have been engaged.

Maybe WotC are already accounting for this, but all the "We're going to need soooooooooooo many physical copies!" stuff sounds more like they're expecting massive uptake. And hell, maybe there will be - we're still, what, 6-12 months out probably, maybe even longer - with a sufficiently good advertising campaign maybe they can turn that sentiment around. But I'm skeptical.

 

[OB1]

I'm not a survey/polling expert, but the sample size seems fine, it's about 10,000x the typical sample size percentage for presidential election polling, for example. In this case, the community of players you play with probably does influence your own style, and therefore the answers the 1 person who takes the time out of 18 or 30 in that community of players does end up reflecting some of the others thoughts.

My bigger concern has always been whether WotC is correcting for bias in their samples for things like play style and also age, race, gender, etc. Given the feedback on the Druid, however, I'd guess that while those biases may slip in during design and closed playtesting, the playtest survey seems to be doing a good job of correcting for it.

 

[Retreater]

I have the opposite problem. I've been told, repeatedly, that we aren't changing editions "ever again." Even after a decade, my friends are still salty over the 3.5E/Pathfinder/4E wreck. We're going to play 5th Edition forever, with or without WotC.

This is a really strange position to be in. People who don't want to play a different game have now the options to a) purchase the 2024 update and keep playing 5e; or b) keep playing the 2014 version of 5e.
Those of us who want something different and actual improvements to the game have no choice but to look at other game systems.
Talking to my good friend last night, he said, "this is the first time in 33 years I haven't been playing D&D."
It kinda hurts to step away from the brand and fandom. There's a lot of history and memories there.

 

[MockingBird]

I have the opposite problem. I've been told, repeatedly, that we aren't changing editions "ever again." Even after a decade, my friends are still salty over the 3.5E/Pathfinder/4E wreck. We're going to play 5th Edition forever, with or without WotC.

Honestly that's where I'm at. I'm heavily invested in 5e, I'm getting older, and this is going to be it for me. If I was in a group of other hardcore rpg fans I might have a different view. I'm the only one in my group who follows it and even participates on forums. To them it's just something we get together and do a couple times a month, like poker.

 

[CleverNickName]

Apparently about 900,000 are involved in the playtest, versus 13 million who have Beyond accounts.

Hm. I wasn't off by much. I'm one gamer out of 18 total, so that would be about 6% (1/18 = 5.55%)

And this estimate of 900,000 playtesters out of 13 million is about 7% (900K/13M=6.92%)