D&D (2024) D&D 5.11 - the time of big change is over

[Marandahir]

Regarding UA survey results and analysis:

I've always wondered about the statistical analysis of their surveys. The sample is a self-selected population of engaged gamers; how do they draw conclusions applicable to the wider public?

I mean, at the end of the day, the whole process seems to be working well for them, given the huge continuing success of the game, but still I'm curious about the methodology.

It’s worth rewatching this D&D Beyond official video from February explaining their analysis of the survey results and how they balance against broader interest: why the Ardling and Jump Action got cut, why Goliath is in, how they parse more than just the numbers, and how they take online discourse and other feedback into account, especially when there’s stark differences between the survey results and the prevalent discourse.

YouTube link to survey results video

 

[Shardstone]

I kinda feel that this is the double edge sword of their A/B testing.

Give a person a glass of cola and ask if they like it. You will probably get a generalized answer about their like or dislike of colas. Give that person two different colas and ask which they like will get you a different answer. The reason is that the person now has a frame of reference to describe the two colas against each other, vs a generalized notion of cola.

The playtest didn't give us what the B was to A/B testing. We're not getting do you like this or that? In some cases, we weren't even asked if we liked A in the first place (subclass progression). We weren't asked which fix for wild shape we preferred, we were given a half-baked template and when people rejected it for a multitude of reasoning, it was all put in the "doesn't spark joy" column. It's why even though I was hesitant about elements of the warlock being a half-caster, I have full throated support for it. Because anything less would give them ammo to pop it into the "no joy" bin.

I hate that we have been conditioned into this

 

[mamba]

I hate that we have been conditioned into this

then don’t, I will mostly give verbal feedback and not rate, because I feel they misinterpret the rating