D&D 5E (2024) The WotC Playtest Surveys Have A Flaw

[Micah Sweet]

I like canons. How big is yours?

24-pound.

 

[jasper]

I have lost 3 gamers due to propose changes and other stuff. I am the only one in my Wednesday group which follow the surveys. And we have maybe six out 40 who follow the surveys on our local facebook page. The numbers could be higher becasue we have 350+ on our Facebook page but I can only get around a 20+ response to topics which get lots of interest. The looks of horror or that stupid on the faces of my Wednesday games are interesting.
I am also going to change the way I answer surveys. I was answering like how well it was going to work as a new edition which is honestly not going to be a new edition just trust us.
Now when I answer the survey it going to be how well I can use the PHB 2024 with PHB 2014.

 

[Ulorian - Agent of Chaos]

I'd also point out that WotC estimates D&D's playerbase at 30m actually playing this edition. Yet the peak survey fill-in was I believe, 40k people according to WotC (correct me if I'm wrong).

Unless my math is failing me, 40k is 0.13 percent of the playerbase (sorry for my earlier decimal place error).

So WotC are actually talking about 70% of 0.13%, so what, 0.091% of the playerbase? That's the approval threshold here.

Whereas your numbers are based on you as 1 of 18 actually interested.

But either way - it's a tiny, insignificant percentage, and the chance of it being representative, given how the survey is being conducted? Non-existent.

The flip side of that coin is that there is a subset of the total player base who has enough knowledge/cares enough about the game to provide meaningful feedback, and these are the folks filling out these surveys. These polls are a slog... only D&D diehards, whose investment in the game will give them the strength to get to the end of the survey, will have their opinions counted. Also, non-diehards who have OCD completionist tendencies

So... the question is: is the focus on D&D 'experts' in these surveys a good or bad thing? Does the sample size/selectivity of this group lead to valuable feedback?

 

[Horwath]

The flip side of that coin is that there is a subset of the total player base who has enough knowledge/cares enough about the game to provide meaningful feedback, and these are the folks filling out these surveys. These polls are a slog... only D&D diehards, whose investment in the game will give them the strength to get to the end of the survey, will have their opinions counted. Also, non-diehards who have OCD completionist tendencies

So... the question is: is the focus on D&D 'experts' in these surveys a good or bad thing? Does the sample size/selectivity of this group lead to valuable feedback?

This.

I also hope that any vote without atleast one sentance(where it is possible) explaining why they voted that way is summarily dismissed.

 

[Nikosandros]

The flip side of that coin is that there is a subset of the total player base who has enough knowledge/cares enough about the game to provide meaningful feedback, and these are the folks filling out these surveys. These polls are a slog... only D&D diehards, whose investment in the game will give them the strength to get to the end of the survey, will have their opinions counted. Also, non-diehards who have OCD completionist tendencies

So... the question is: is the focus on D&D 'experts' in these surveys a good or bad thing? Does the sample size/selectivity of this group lead to valuable feedback?

I consider myself quite invested in D&D, but those surveys are excessive for me. I've stopped filling them.

 

[Ruin Explorer]

So... the question is: is the focus on D&D 'experts' in these surveys a good or bad thing? Does the sample size/selectivity of this group lead to valuable feedback?

Historically, looking at videogames which undertook this kind of in-depth self-selecting surveying? Which is the only comparison point I'm aware of.

It's good for one thing - point specific pain points with specific abilities/spells/etc. that the people in charge of the game might not be aware of. Sometimes that's misleading, because it's a pain point only for the elite, or because the elite want to continue to abuse a broken ability, but more often than not, it's general issue, when we're talking about like the specific stuff. That's the upside of the "rate every ability of this class individually" thing - usually the ultra-hardcores will somewhat align with the playerbase in general on specific. You do have to be wary because there are exceptions, but generally.

What's it's very bad for, unfortunately, is the determining the general direction of a game, of classes, of larger issues in general, because the same exact people who are genuinely pretty good at picking out pain points, and fantastic axe-grinders with weird ideas that they often share with each other, but not with the bulk of the people playing the game. They're also ultra-conservative, typically, in a way that most players are not. You see this with MMOs that have undertaken these sort of in-depth self-selecting surveys - the information they get from them will often conflict with more straightforward surveys which reach a larger number of players, or surveying which isn't self-selecting but put out by the company.

Like, if you were running an MMO, and the main thing you wanted, was to retain the most serious raiders and PvPers, and you didn't care about the 95% who were the bulk of your paying customers, surveying like this and following their feedback. But that's focusing on 5% of the market at the cost of the rest. History suggests when you're a market leader, that will actually work for a while, whether you're EQ or WoW or whatever, but then it stops working.

A lot of companies did some fairly extensive surveying and research into what MMORPG players said they wanted (a lot of them with self-selecting surveys) back in the '00s, and this lead to quite a lot of failed MMORPGs. Not even ones that went F2P and lived - ones that outright died. I don't think the same will happen with a market leader like D&D but I also don't think anything good will happen from listening to ultra-nerds (which 100% includes me) about the direction of your game, and not listening to Jonny or Jenny who plays D&D but less seriously. Anyway, this kind of "What do people SAY they want?" stuff lead RPGs like Vanguard, WildStar, and a bunch of near-FFA PvP MMORPGs, all of which ended in tears. Because what the serious, in-depth, detailed-oriented, hmmmm how to put this... often neurodiverse (I say that as a seriously neurodiverse person) people say they want is NOT, absolutely NOT the same as "what people who play these games actually like". There's almost no finer evidence of that than FFXIV's success. It's basically done everything hardcore people said they didn't want in an MMORPG. WoW has also painfully and with difficulty moved away from listening to a tiny group of ultra-hardcore people, and they only really made the break when all the old guard had to leave or got fired in the sex abuse scandal and related fallout, and now we have Dragonflight, which is easily the best WoW expansion in over a decade - and it's primarily younger people, blue-haired people and so on, designing it - and designing it for people who actually play WoW!

The flip side of that coin is that there is a subset of the total player base who has enough knowledge/cares enough about the game to provide meaningful feedback, and these are the folks filling out these surveys.

This kind of attitude is part of the problem and how companies get into trouble with this kind of surveying.

Because conflating "enough knowledge" and "cares enough" is extremely dangerous, and assuming that he hardcore people have that knowledge, rather than just specific axes they grind, is dangerous too.

As I said, these kind of people can be useful for specific pain points and issues, but for general directions of classes? For how your game should look? For what people want in a broader sense? No. Absolutely not. These people are not reliably representative of the bulk of people playing. The most recent example is the apparent insistence that the entire arcane/divine/primal spell list concept be abandoned solely for the sake of Wizards. That is the sort of thing hardcore axe-grinders obsess about, and can't see the bigger picture about. One of the big things with MMORPG design and the like was designers realizing that, at some point, they have to be able ignore feedback and say "No players, actually you are wrong, and we are right".

A specific example of that would be WoW's obsession with high-end raiding. The feedback they were getting was that it was everything, and the devs actually were aligned with that feedback because they were from a similar background to the people giving it. Eventually, however, the CEO of the company had to step in, because WoW's own metrics (metrics sadly unavailable to TT RPGs) showed with the last high-end raid they'd released, what % of the playerbase had even seen the inside of it?

0.5%

And yet their design and class-balancing efforts for months had been focused on that raid and its environs. And the self-selecting feedback they had said they were right to be doing that.

Anyway, I could go on and on, but the point is, it's worth having that feedback for pain points and specific issues, but as a game designer, you need to able to say "Yeah that feedback says X, but what we actually need to do is Y". Frankly if WoW hadn't been able to do that, The Burning Crusade would have represented it's peak population, and it'd been F2P for years by now.

 

[mamba]

I have lost 3 gamers due to propose changes and other stuff. I am the only one in my Wednesday group which follow the surveys. And we have maybe six out 40 who follow the surveys on our local facebook page.

so 3 out of 6 stopped playing because of proposed changes, is that what you are saying?

Why would they drop out now?

 

[mamba]

This.

I also hope that any vote without atleast one sentance(where it is possible) explaining why they voted that way is summarily dismissed.

if they did that, the text should be mandatory, you cannot move on without filling it.

Also, does ‘I like it better’ count? If so, there is no point requiring a text

 

[Atomoctba]

Someone who does have a perfectly good core set but buys a new one.

I know at least three people that will buy the new one no matter if they think they are good or bad or if should have more/less changes. They will buy that only for "collector's urge" (in a lack of better term). See, the new books have attached to them the fact they will be the 50th anniversary of D&D. So, some people will buy them because that if anything else. Even to just lie them on the shelf.

 

[Ruin Explorer]

I know at least three people that will buy the new one no matter if they think they are good or bad or if should have more/less changes. They will buy that only for "collector's urge" (in a lack of better term). See, the new books have attached to them the fact they will be the 50th anniversary of D&D. So, some people will buy them because that if anything else. Even to just lie them on the shelf.

There are absolutely people like that out there.

With some products, that's even the majority of people they sell to.

I think, based on comments WotC have been making over the last 20 years, including recently, that despite their attempts to push D&D into become a "lifestyle" brand (and thus more like that, customer-wise), it's still a pretty small minority of D&D purchasers who operate that way. Also note the full-branded "anniversary edition" 5E books have apparently been delayed to 2025 (rather hilariously). Of course I suspect anyone wanting those will likely also buy the 2024 non-anniversary ones.