D&D (2024) The WotC Playtest Surveys Have A Flaw

[CleverNickName]

What is the remedy? Force folks that don't want no edition change no more to participate?

This is the crux of the whole topic, I think. What do we do about it?

I think there are a couple of things we could do.

Remember Who We Speak For
As only about 6% of the player base, each survey response is speaking for 16, maybe 17 other people! So when you respond, remember that you are also speaking for your friends and fellow players at the table. Talk to them about the playtest, take note of the things they like, and try to keep them in mind when you're filling out the survey.

My friend Doug, for example, hates warlocks. He thinks they are a blight on the D&D landscape, from their goth edgelord energy they bring to the lore, to their spammy way of casting spells, he wants them removed from the game. Now obviously he's wrong, warlocks are undeniably awesome. So when I filled out the playtest survey, I spoke highly of the warlock and all the things that I liked about it--but I also mentioned my friend Doug (by name!) in the comments and voiced his opinion as well, even though it conflicted with my own.

And I'm also speaking for my nieces and nephews, who are just getting into the hobby and are really excited about owlbear druids and princesses with fairy wings and playable unicorns. They don't give a fig about "bounded accuracy" and "action economy" or the "martial vs. caster disparity," they only care about freedom and imagination. I need to make sure the hobby doesn't leave them behind, that twenty years from now they will be able to pick up a D&D book and smile and remember being a kid. So I try to look at the game through their eyes, and remember that fun must always be paramount when I'm filling out that survey.

Temper Our Expectations
Remember that we are only 6% of the player base. (Sorry to keep harping on it, but it bears repeating.) So when Wizards of the Coast talks about "70 Percent Approval!" they mean 70% of that 6%. Their 70% approval is only about 4.2% of the player base...and that isn't going to win any elections. It would be folly to base the direction of their flagship, breadwinning product on such a low sample size, right? So it's safe for us to assume that they are also listening to other sources of feedback: shareholders, sure, but also their own playtesters, and professional game developers, experienced writers, professional mathematicians and economists...it would be foolish of them not to.

So we should temper our expectations a bit, and not get inflamed with outrage when we don't get everything we wished for. I imagine that when the next product is released, we will be able to see our influence and feedback to some degree...but it'll also be pretty clear that we were not the ones in the driver's seat. And we never were.

EDIT: What @Vael said.

If the survey was the only thing WotC was doing, I'd be more concerned. But they have internal playtesters, focus groups, other ways to acquire data.

I'm the only one in my IRL DnD circles that even participates on a board like ENWorld, so I know I'm the outlier who's more plugged into what's going on, and one of a few that has even looked at the playtest materials (I offered to let players use them in the last short campaign I ran and everyone just went back to the 2014 books or DnD Beyond).

TBH, I think that makes me the outlier. So I expect some analysis of the unsurveyed. That's probably why the threshold for change is 70%

 

[Ruin Explorer]

I'm just going to throw this out there- while I am not particularly happy with all of the changes, either, I think that it might just be because ... the issue isn't the playtesters, but us.

Maybe we're the weird ones? And the desired demographics for the game probably isn't a bunch of older gamers that spend their time on an internet forum complaining about 5e?

So you might be right! You may well be 100% right.

But here's the problem. We're essentially the ones deciding the fate of the stuff that's going into the surveys. That 0.01% is I would suggest pretty much undoubtedly more "us" than any other category of D&D players. If we remember the chart where we're like 10% of D&D players, I would guess that people 40+ are probably 50% or more of the people filling in the survey. And that they're probably very online, axe-grind-y D&D players like us, not representative of the bulk of D&D players.

So "we" (in a sense) are the ones who keep going "Lame idea, dump it!", and maybe that's actually a great idea that the 20+ million younger and more diverse and less axe-grind-y players would like, but the 20-40k people (or less!) people filling in the survey gave it 50% instead of 70% and so now that idea is OUT!

Either way, it kind of doesn't matter, the surveys are causing "us" to be rejecting this stuff!

 

[jmartkdr2]

Copied from the other thread:

I, for one, don't answer the surveys. I got turned off pretty early in the playtest process and just no longer care what they do with it.

So even among grognards who go online to discuss the future of DnD - they're getting a sample. Possibly a good sample of that small group, but still.

 

[Snarf Zagyg]

So you might be right! You may well be 100% right.

But here's the problem. We're essentially the ones deciding the fate of the stuff that's going into the surveys. That 0.01% is I would suggest pretty much undoubtedly more "us" than any other category of D&D players. If we remember the chart where we're like 10% of D&D players, I would guess that people 40+ are probably 50% or more of the people filling in the survey.

They ask for ages, so unless everyone is lying, I assume that they are doing rudimentary statistical analysis to ensure that this isn't the AARP edition.

Not that there's anything wrong with that. I, for one, look forward to restoring Save vs. Rod, Staff, and Wand.

 

[Zaukrie]

No pollster thinks their sample is "everyone in the room." I really don't think most people here have experience doing this kind of work.....

 

[Ruin Explorer]

Remember that we are only 6% of the player base. (Sorry to keep harping on it, but it bears repeating.) So when Wizards of the Coast talks about "70 Percent Approval!" they mean 70% of that 6%. Their 70% approval is only about 4.2% of the player base...and that isn't going to win any elections.

Not making this political but this reminds me of the situation with Prime Ministers in the UK. In the UK, once a party is in power, they can appoint whoever they want, however they want, as PM. With the current government, they have a bizarre method where the members of the party vote, and no-one else - and that's like, 170k people determining the fate of the entire country.

It's not great. And it's not great with D&D.

I'd also point out that WotC estimates D&D's playerbase at 30m actually playing this edition. Yet the peak survey fill-in was I believe, 40k people according to WotC (correct me if I'm wrong).

Unless my math is failing me, 40k is 0.13 percent of the playerbase (sorry for my earlier decimal place error).

So WotC are actually talking about 70% of 0.13%, so what, 0.091% of the playerbase? That's the approval threshold here.

Whereas your numbers are based on you as 1 of 18 actually interested.

But either way - it's a tiny, insignificant percentage, and the chance of it being representative, given how the survey is being conducted? Non-existent.

 

[Ruin Explorer]

They ask for ages, so unless everyone is lying, I assume that they are doing rudimentary statistical analysis to ensure that this isn't the AARP edition.

Not that there's anything wrong with that. I, for one, look forward to restoring Save vs. Rod, Staff, and Wand.

But that's strongly incompatible with their "70% threshold" claim.

They can't really have it both ways. Either you can weight stuff and avoid AARP edition, or you can use an absolute threshold, and get damned or helped by that. Maybe AARP edition is fine.

Now I know they're not compelled to follow the 70% threshold, but their own messaging, the stuff Crawford has said himself is quite persuasive to the opinion that they basically are doing exactly that in most cases! It's like, the alternative is that Crawford is lying, and I just don't see him as that kind of machiavellian, quite the contrary.

Also, to be fair to us grogs, most of aren't "RETVRN" types, who wish 5E would turn into an OSR game or something. I think most grogs are fairly reasonable, I just don't think they're at all representative, particularly re: classes. I'm really skeptical that something age-weighted would be causing them to redo the entire magic approach so Wizards can have the biggest spell list again.

 

[Scribe]

My friend Doug, for example, hates warlocks. He thinks they are a blight on the D&D landscape, from their goth edgelord energy they bring to the lore, to their spammy way of casting spells, he wants them removed from the game. Now obviously, he's absolutely wrong, warlocks are far better than most other spellcasting classes in my opinion. So when I filled out the playtest survey, I spoke highly of the warlock and all the things that I liked about it--but I also mentioned my friend Doug (by name!) in the comments and voiced his opinion as well, even though it conflicted with my own.

I dont trust Wizards any further than I can throw their Ivory tower, why would I intentionally muddy the water and say 'Yeah I feel this, but this other guy feels a different way.'

I dont understand how that is going to do anything but more firmly plant the fence post they are straddling already...well...

Argue for what you personally want. The idea that they can satisfy you, a Warlock enjoyer, and your buddy who hates everything about them, is just not going to happen in my view.

 

[OB1]

For an opt in survey, you need a sample size of about 0.038% to get within 5% margin of error, WotC sample size is about 4x bigger than that. So with a 70% threshold, WotC should have high confidence that a minimum of 65% of the player base has approved of something. That put's it in 2/3rds consensus for change.

 

[CleverNickName]

I dont trust Wizards any further than I can throw their Ivory tower, why would I intentionally muddy the water and say 'Yeah I feel this, but this other guy feels a different way.'

I dont understand how that is going to do anything but more firmly plant the fence post they are straddling already...well...

Argue for what you personally want. The idea that they can satisfy you, a Warlock enjoyer, and your buddy who hates everything about them, is just not going to happen in my view.

I think the bigger problem is the idea that they can satisfy everyone, period, with any decision they make in the game. We have proven time and again that the only thing we love more than D&D, is arguing about D&D.

But I'm not trying to win an argument with Doug, I'm trying to make sure the final product is enjoyed by the most people possible. Assuming that "if they do what I want, it will be a success" is kinda weird.