D&D (2024) The WotC Playtest Surveys Have A Flaw

[mamba]

Kinda depends on where you want to go. If they are being conservative then they increase the approval threshold until it makes the systematic errors irrelevant.

irrelevant in what way? That you let nothing ‘bad’ in? Sure, you can do that, but increasing the threshold will also keep some ‘good’ out that you would have wanted

Depending on how far you go, you are no longer improving in the direction the majority favors, you are only avoiding disaster. Some very clearly preferred options can still make it in, but the main goal at that point is avoiding mistakes, not making improvements

 

[UngainlyTitan]

The middle ground is they may not be incompetent, but they may not be able to get the data they need. This happens all the time in school districts. They have some good statisticians, yet because of the community's lack of responses, they get many things wrong: projections, percentages for classes, etc.
So it may be that WotC pays these statisticians, and the stats people can't acquire the information they need. And then, not wanting to hand over nothing, they hand over something with a mistake. And I know they will give a confidence level, but even that has been wrong at times. Like, very wrong.

I really do not believe that lack of response is the issue. They also have sales data and usage data from D&DBeyond, probably also from Roll20 and FantastyGrounds.

 

[UngainlyTitan]

irrelevant in what way? That you let nothing ‘bad’ in? Sure, you can do that, but increasing the threshold will also keep some ‘good’ out that you would have wanted

Depending on how far you go, you are no longer improving in the direction the majority favors, you are only avoiding disaster. Some very clearly preferred options can still make it in, but the main goal at that point is avoiding mistakes, not making improvements

They don't care about "good" or "bad". They have a product that is selling well enough. They also know that adding popular features will keep it fresh in the market.
If they are not certain that the new thing is not more popular than the current thing then they default to stick with the current thing.
If "good" ideas get left out but the process, well, thems the breaks.
This strategy will remain as long as it is commercially successful.

 

[Ruin Explorer]

They don't care about "good" or "bad". They have a product that is selling well enough. They also know that adding popular features will keep it fresh in the market.
If they are not certain that the new thing is not more popular than the current thing then they default to stick with the current thing.
If "good" ideas get left out but the process, well, thems the breaks.
This strategy will remain as long as it is commercially successful.

This is a surprising precarious strategy with games, as videogames, particularly MMORPGs, show. Especially if your data on what is "popular" isn't derived from actual gameplay, but rather from a small group of fans. Luckily most videogames now have much more direct ways to get metrics, but there was an era, particularly in the '00s, before in-game metrics were as well developed (or where they'd just been overlooked), where a lot devs did rely on small groups of fans for feedback, and... it went well until it didn't, as it were.

At the risk of sounding like a LotR character, there will be a day when D&D starts to lose popularity, and on that day, it is quite likely that one of the reasons for that is, one of the major reasons, is WotC's utter reliance on these tiny, unfocused rate rate rate rate rate surveys failing to account for the desires of the bulk of the fans.

One interesting thing will be, when the 3D VTT comes online (assuming it doesn't explode), that will finally give WotC real metrics. Not metrics of how people play D&D generally, but metrics of how people play the 3D VTT, and I would be very surprised if that doesn't profoundly impact D&D's design over the next decade.

 

[UngainlyTitan]

This is a surprising precarious strategy with games, as videogames, particularly MMORPGs, show. Especially if your data on what is "popular" isn't derived from actual gameplay, but rather from a small group of fans. Luckily most videogames now have much more direct ways to get metrics, but there was an era, particularly in the '00s, before in-game metrics were as well developed (or where they'd just been overlooked), where a lot devs did rely on small groups of fans for feedback, and... it went well until it didn't, as it were.

At the risk of sounding like a LotR character, there will be a day when D&D starts to lose popularity, and on that day, it is quite likely that one of the reasons for that is, one of the major reasons, is WotC's utter reliance on these tiny, unfocused rate rate rate rate rate surveys failing to account for the desires of the bulk of the fans.

One interesting thing will be, when the 3D VTT comes online (assuming it doesn't explode), that will finally give WotC real metrics. Not metrics of how people play D&D generally, but metrics of how people play the 3D VTT, and I would be very surprised if that doesn't profoundly impact D&D's design over the next decade.

First I am in disagreement with you that the sample sizes are too small to provide useful data.
Neither of us have any idea if they do other market research but you seem to be assuming incompetence.
Unless the business starts failing we will not know if you are correct.
Finally, you are quite correct that the VTT, if it gains market share will be a rich source of data.

 

[Micah Sweet]

irrelevant in what way? That you let nothing ‘bad’ in? Sure, you can do that, but increasing the threshold will also keep some ‘good’ out that you would have wanted

Depending on how far you go, you are no longer improving in the direction the majority favors, you are only avoiding disaster. Some very clearly preferred options can still make it in, but the main goal at that point is avoiding mistakes, not making improvements

I can certainly agree that WotC's goal with 5e is not to make improvements.

 

[Ruin Explorer]

Neither of us have any idea if they do other market research but you seem to be assuming incompetence.

I am assuming they're not lying re: the 70% approval threshold.

Which leads directly to, if not incompetence, then certainly something that could very easily look like it.

I can certainly agree that WotC's goal with 5e is not to make improvements.

It's certainly not the primary goal, it's a "nice to have".

The primary goal seems to be "don't rock the boat"/"don't kill the golden goose".

The thing is, I don't think there was really a serious risk of either from any design changes/UAs that 5E has ever had. I feel like they could have gone the wildest, most risky choice possible in almost every single UA over the years, completely ignored surveys, and I'm pretty sure 5E would be just as successful as now, if not slightly more successful. For example, what if they'd put in a properly-balanced and playtested version of the Mystic (really a Psionicist)? Would the game be less successful or popular? No. There's just no way. No-one would have quit. There'd probably a widely-loved Critical Role character who was a Mystic, and it'd be like, one of the mid-popularity classes, or at least above the bottom ones (sorry Druids - you're actually pretty good!). Some tiny portion of groups would ban it, sure, but that's the same sort of assorted races, classes, subclasses.

Even with these 2024 UAs, the only stuff I've seen that would have actually have caused serious ructions and maybe dented sales by like, 5%, would have been the proposed Druid and Warlock changes. Like the Aardlings, what if they'd pushed them through? No-one would have quit or been seriously mad, or so few people that they were balanced out by the dear Furries who were delighted by it. I guess if they'd pushed the "you are really one race" hybrid thing through that would have caused some real blowback, especially outside the US, but I'm not sure they're not going to, and it's different in nature to the other issues the UAs have covered.

 

[tetrasodium]

and if they do not agree you do what?

Checks are cashed and maybe in a few years someone decides to write their thesis about the management practices that allowed that enormous amount of money to be spent pointlessly trying to prove that a questionable survey was anything but.

This is a surprising precarious strategy with games, as videogames, particularly MMORPGs, show. Especially if your data on what is "popular" isn't derived from actual gameplay, but rather from a small group of fans. Luckily most videogames now have much more direct ways to get metrics, but there was an era, particularly in the '00s, before in-game metrics were as well developed (or where they'd just been overlooked), where a lot devs did rely on small groups of fans for feedback, and... it went well until it didn't, as it were.

At the risk of sounding like a LotR character, there will be a day when D&D starts to lose popularity, and on that day, it is quite likely that one of the reasons for that is, one of the major reasons, is WotC's utter reliance on these tiny, unfocused rate rate rate rate rate surveys failing to account for the desires of the bulk of the fans.

One interesting thing will be, when the 3D VTT comes online (assuming it doesn't explode), that will finally give WotC real metrics. Not metrics of how people play D&D generally, but metrics of how people play the 3D VTT, and I would be very surprised if that doesn't profoundly impact D&D's design over the next decade.

I don't think that the vtt will give as much data as you suggest without being impossibly difficult to prep for a session or so limited that prep amounts to picking adventure abc

 

[Ruin Explorer]

I don't think that the vtt will give as much data as you suggest without being impossibly difficult to prep for a session or so limited that prep amounts to picking adventure abc

Why do you say that, specifically? I'm unsure on what would prevent them gathering metrics here.

 

[tetrasodium]

Why do you say that, specifically? I'm unsure on what would prevent them gathering metrics here.

Couple reasons rooted in the fact that I've used a vtt with attached display for years of in person GM'ing now.

The. Vtt use splits in two ways. Firstly you have a pretty battle mat replacement that might sport some dynamic lighting and fow, all of the game is still handled outside the vtt even if sometimes parts are still done with the vtt itself. While that battlemat replacement is more than functional enough and looks amazing it still adds some prep work on top of the usual gm prep.at the other end you have GMs who go woo out and try to configure the vtt to handle everything the vtt could manage or automate... That's an extreme timesink that collapses back to a battlemat replacement as soon as the players try something unexpected or decide to blaze their own path. In the middle is a little of each as convenient.

You can get around both of those and get more data if you build the vtt to be played like a video game with loadable adventures rather than like d&d. None of those are really going to give you great data about actual d&d play without some agi listening in to crunch the conversation for analysis though