D&D 5E (2024) A critical analysis of 2024's revised classes

[ECMO3]

By definition it wasn’t a representative survey.

I no longer believe this due to the new information that came to light.

That link someone posted by Mike Mearls above (the only one on this thread who would actually know) indicates it was not dedicated players that took the survey; something I was originally willing to accept at face value but no longer am willing to accept after reading Mearls post.

Being a non-representative survey does mean it’s a flawed to draw any conclusions about the whole population from it.

This statement would be true if it was a survey that did not represent the population, which I did agree with but no longer do agree with.

However, even if I still accepted this premise - saying it is flawed to draw any conclusions from it is fundamentally different than saying the results are skewed or incorrect.

 

[TwoSix]

Whatever they did, it was probably already better research than any of us did.

Who here has done actual surveys to come to their conclusion?

Fortunately, I don’t need a survey to know that I know better than everyone else.

 

[Zardnaar]

Whatever they did, it was probably already better research than any of us did.

Who here has done actual surveys to come to their conclusion?

This. I dont think the surveys are perfect. I dont expect them to be.

They are good at figuring out what people hate. That's all they need to know.

 

[mamba]

Whatever they did, it was probably already better research than any of us did.

Given the methodology used, I'd say their 'research' was not much better than a coin toss at identifying improvements

Who here has done actual surveys to come to their conclusion?

Given the accuracy of their surveys (see above), I do not need one to be about as accurate

 

[Sulicius]

Given the methodology used, I'd say their 'research' was not much better than a coin toss at identifying improvements


Given the accuracy of their surveys (see above), I do not need one to be about as accurate

Yeah I don’t know how to deal with this level of pessimism and willful ignorance.

Or are you being sarcastic?

 

[tetrasodium]

Yeah I don’t know how to deal with this level of pessimism and willful ignorance.

Or are you being sarcastic?

Self selected surveys like the ones they did are so notoriously likely to be skewed by an overrepresented unusually passionate minority among possible respondents that they teach it in college courses related to polling and statistics. All of that is before you consider the impact of statistical insanity like the time Crawford told us all that when they saw things like a class getting high marks on individual features but poor in overall class grade as a whole they assumed respondents forgot what they voted and used the higher value. Another great example was the time that dndshorts put out a "there's a problem with the new warlock" video Esther hysterically calling for survey brigading with a video that got more upvotes than Crawford later bragged about that survey getting as reason for changing course

 

[Sulicius]

Self selected surveys like the ones they did are so notoriously likely to be skewed by an overrepresented unusually passionate minority among possible respondents that they teach it in college courses related to polling and statistics. All of that is before you consider the impact of statistical insanity like the time Crawford told us all that when they saw things like a class getting high marks on individual features but poor in overall class grade as a whole they assumed respondents forgot what they voted and used the higher value. Another great example was the time that dndshorts put out a "there's a problem with the new warlock" video Esther hysterically calling for survey brigading with a video that got more upvotes than Crawford later bragged about that survey getting as reason for changing course

What is your proof that they are self selected? What do you belief? And why do you believe it?

I never heard anything about Crawford saying players forgot what they voted, or maybe that’s from before I played.

I just shared first-hand experience of Mike Mearls about this, and I see nothing but refutals without proof.

 

[tetrasodium]

What is your proof that they are self selected? What do you belief? And why do you believe it?

It's literally the type of survey they did with the public surveys. Here's a very simple article about self selected surveys

https://www.scribbr.com/research-bias/self-selection-bias/

I never heard anything about Crawford saying players forgot what they voted, or maybe that’s from before I played.

He brought it up in one of the onednd videos

I just shared first-hand experience of Mike Mearls about this, and I see nothing but refutals without proof.

 

[mamba]

Yeah I don’t know how to deal with this level of pessimism and willful ignorance.

Or are you being sarcastic?

no, that is my assessment of their methodology and their handling of feedback. Their approach is about as good at identifying improvements and then sticking to working on them as tossing a coin would be.

Where it considerably beats a coin toss is at identifying total duds and discarding them. It is very reliable there, but pretty useless for anything else. I have been saying this during the UA phase too.

I am honestly surprised they stuck with such a flawed methodology for so long. The only conclusion I can draw is that they do not care about identifying improvements, only about weeding out complete duds, and that the UA mostly is about marketing / engagement / raising awareness about the new product, rather than actually improving the game

 

[mamba]

All of that is before you consider the impact of statistical insanity like the time Crawford told us all that when they saw things like a class getting high marks on individual features but poor in overall class grade as a whole they assumed respondents forgot what they voted and used the higher value.

and don’t forget the time where they dropped something that got more than the 70% threshold vote regardless of that, because some comments really did not like the idea

The main issue is even deeper, it is that there is no good way to actually communicate intent (outside of always leaving a comment), so they do not know what me voting 2 actually means and I have no idea what I would have had to vote for them to react in the way I would have wanted them to.

Add to that the 70% threshold for whether people supposedly like a feature on its own merits when in reality no one is evaluating it on its own but comparing it against what exists today and that the feedback is supposed to ignore balancing when the vast majority doesn’t and you get meaningless numbers that are being interpreted randomly, and yet somehow expect that this leads to improvements

I agree that it identifies what is overwhelmingly unpopular, for anything else it is meaningless