D&D (2024) The WotC Playtest Surveys Have A Flaw

[Clint_L]

This thread is forked from another, unrelated thread. Things started to drift off-topic there, but I felt it's a good topic for discussion.


This is a big one.

I belong to three different gaming groups. And of those 18 people in total, I'm the only one who is following the development of the game at all. I'm the only one who knew that Wizards of the Coast is working on a new edition rules revision, and my fellow gamers get really defensive when I mention it. One guy will actually growl at me every time I bring up the playtest, "We are not changing editions again!"

For those still wondering why WotC isn’t interested in publishing a whole new edition, here’s why.

 

[Retreater]

Maybe WotC are already accounting for this, but all the "We're going to need soooooooooooo many physical copies!" stuff sounds more like they're expecting massive uptake. And hell, maybe there will be - we're still, what, 6-12 months out probably, maybe even longer - with a sufficiently good advertising campaign maybe they can turn that sentiment around. But I'm skeptical.

If they're expecting it to sell like the 5e books, I think we'll be seeing those in Ollie's Bargains by the start of the 2024 holiday shopping season.

 

[payn]

What is the remedy? Force folks that don't want no edition change no more to participate?

 

[Ruin Explorer]

WotC doesn’t need to publicize how they do focus groups in the same way they need to publicize the survey, so I wouldn’t make absence of evidence into a strong evidence of absence.

I would agree that it's not strong evidence of absence. However, my experience with games companies when they do this is that they do tend to crow about, and again, with Crawford, I just don't believe for a second he'd have just never, ever, not once in this process mentioned that they were doing that. His team too. Someone would have said something. They'd accidentally scooped up a TikToker or six and they'd have talked about it (that generation does not know what an NDA is).

So I'm not saying we can be certain they're not doing it. I do however think it's very unlikely they're doing it.

Surveys are easy to weight on demographics if you know the base demographics, and there’s probably some proxies in there. The qualitative responses will likely contain insights unrelated to that. But also this means that quoting raw approval numbers isn’t very illuminating.

And that's the weird thing - they could be weighting, but they keep coming back, entirely voluntarily, to that magic 70% threshold, and not once have they said "Wellll, younger players preferred this, but older players preferred that" or anything to suggest that demographics even exist to them.

 

[Vael]

If the survey was the only thing WotC was doing, I'd be more concerned. But they have internal playtesters, focus groups, other ways to acquire data.

I'm the only one in my IRL DnD circles that even participates on a board like ENWorld, so I know I'm the outlier who's more plugged into what's going on, and one of a few that has even looked at the playtest materials (I offered to let players use them in the last short campaign I ran and everyone just went back to the 2014 books or DnD Beyond).

TBH, I think that makes me the outlier. So I expect some analysis of the unsurveyed. That's probably why the threshold for change is 70%

 

[Ruin Explorer]

What is the remedy? Force folks that don't want no edition change no more to participate?

For WotC to do proactive research on a decent scale. That costs money, it requires organisation, and so on. Huge money compared to getting people to fill in surveys on a website you already own, but tiny money compared to how much D&D makes. The trouble is, WotC doesn't seem interested in investing in that kind of thing.

I'd say it's significantly too late in the process now. Maybe for next edition, or the next half edition or whatever?

 

[Clint_L]

If they're expecting it to sell like the 5e books, I think we'll be seeing those in Ollie's Bargains by the start of the 2024 holiday shopping season.

These are 5e books, and they aren’t trying to juice sales, though I’m sure sales will spike to some degree, they are trying to maintain sales over the long run.

 

[Parmandur]

Hm. I wasn't off by much. I'm one gamer out of 18 total, so that would be about 6% (1/18 = 5.55%)

And this estimate of 900,000 playtesters out of 13 million is about 7% (900K/13M=6.92%)

Thing is, they might still be able to get a representative sample out of that still, particularly since the threshold for any change is at 70%.

 

[Ruin Explorer]

But they have internal playtesters, focus groups, other ways to acquire data.

I don't see any evidence at all that they have significant use of focus groups. They should do - they should have tons, but they ever talked about them? They're easy to spin very positively.

Looking online, the only thing I can find is that WotC may have done focus groups on D&D players - but not about D&D - about cross-marketing synergies - i.e. what other brands D&D players think are cool.

Internal playtesters, I'm sure exist, but if the material we're getting in the UAs is the result of internal playtesting, I have to say, I think they're probably not very helpful lol. And probably a weird bunch.

And what other ways are they acquiring data? D&D Beyond? We know that, but according to WotC's own accounts of this process, they're pretty much relying on the surveys and nothing else. I'm not actually sure how much useful data they could derive from Beyond. Some I'm sure, but to inform the design of a new edition? Hmmm.

That's probably why the threshold for change is 70%

No, we know why it is, Mearls said so back in the day - it's because it's an overwhelming majority.

And they felt that once 5E, an apology edition, was in place, that was what they needed in order to justify changing anything at all.

 

[Snarf Zagyg]

Internal playtesters, I'm sure exist, but if the material we're getting in the UAs is the result of internal playtesting, I have to say, I think they're probably not very helpful lol. And probably a weird bunch.

I'm just going to throw this out there- while I am not particularly happy with all of the changes, either, I think that it might just be because ... the issue isn't the playtesters, but us.

Maybe we're the weird ones? And the desired demographics for the game probably isn't a bunch of older gamers that spend their time on an internet forum complaining about 5e?