D&D 5E (2024) A critical analysis of 2024's revised classes

Sulicius said:

What is your proof that they are self selected?

Click to expand...

the fact that they are self selected, any survey you post on the internet for people to come and fill out is. That basically is the definition of self-selected

mamba said:

the fact that they are self selected, any survey you post on the internet for people to come and fill out is. That basically is the definition of self-selected

Click to expand...

Ah, my bad, I misunderstood.

mamba said:

no, that is my assessment of their methodology and their handling of feedback. Their approach is about as good at identifying improvements and then sticking to working on them as tossing a coin would be.

Where it considerably beats a coin toss is at identifying total duds and discarding them. It is very reliable there, but pretty useless for anything else. I have been saying this during the UA phase too.

I am honestly surprised they stuck with such a flawed methodology for so long. The only conclusion I can draw is that they do not care about identifying improvements, only about weeding out complete duds, and that the UA mostly is about marketing / engagement / raising awareness about the new product, rather than actually improving the game

Click to expand...

I think the true reason is far simpler - less flawed methodology requires a far more elaborate setup and many more moving parts than they could hope to handle, especially if they have the typical deadlines for "make a new edition".

Seriously, think about how many people you would need to make a representative sample! How do you get those "casuals" to possibly even do an actual playtest? How valuable is feedback based just on reading some rule suggestion actually? If it requries play-test, how long until survey participants have organized a group to play? How many factors are actually influencing the outcome of such a play-test that might be unrelated to the topic of the survey?
How long are they supposed to wait until that feedback is there, and how do they control for all those variables and factors?!
That's almost as complicated at testing new medical drugs! With far lower stakes.

Mustrum_Ridcully said:

I think the true reason is far simpler - less flawed methodology requires a far more elaborate setup and many more moving parts than they could hope to handle, especially if they have the typical deadlines for "make a new edition".

Click to expand...

not really, all it does need is better options to answer, to tell the difference between ‘do not like it’ and ‘like it, but needs some work’ rather than having WotC guessing which of the two it is.

Apparently they now have those with their new red/yellow/green approach (instead of 1/2/3/4), after all the 2014 and 2024 polls… no idea why it took them this long to get something this fundamental addressed

Ideally there would also be clearer communication of what they are looking for (e.g. ignore balance), or a more realistic approach on their end (we know people will compare against what they have today instead of evaluating it on its own, so the poll should be based on that instead of on a fantasy, red/yellow/green helps there too)

Mustrum_Ridcully said:

Seriously, think about how many people you would need to make a representative sample!

Click to expand...

a lot fewer than they have, this is the least of my concerns. I not once used this as a reason for why their polls are flawed

Mustrum_Ridcully said:

How valuable is feedback based just on reading some rule suggestion actually? If it requries play-test, how long until survey participants have organized a group to play?

Click to expand...

given that they are not concerned about balance, I’d say about as good as from someone who played it

Mustrum_Ridcully said:

That's almost as complicated at testing new medical drugs!

Click to expand...

it really isn’t

I kinda laugh when people claim polls are flawed. Biggest D&D edition ever. Translation is dont like something and got out voted.

People aren't really voting on good mechanics as ENworld would define them. They're voting for good enough and stuff they dont like/hate.

ENworlds out to lunch here. Not the polls.

tetrasodium said:

All of that is before you consider the impact of statistical insanity like the time Crawford told us all that when they saw things like a class getting high marks on individual features but poor in overall class grade as a whole they assumed respondents forgot what they voted and used the higher value.

Click to expand...

Are you sure that's what was said? I remember a video where Crawford mentioned that exact same scenario and they were making an entirely different assumption. That it made them look at it as individually these features are hitting the mark, but collectively the class is missing something fundamental. That the features weren't a problem, they just weren't enough.

Zardnaar said:

I kinda laugh when people claim polls are flawed. Biggest D&D edition ever. Translation is dont like something and got out voted.

Click to expand...

not really, I don’t see much correlation between 5e’s success and their polling ability. Also, I like 5e and I still think their polling methodology is about as bad as it can be

TwoSix said:

Fortunately, I don’t need a survey to know that I know better than everyone else.

Click to expand...

That is not what I put in my survey

Regardless of the effectiveness of the survey, I think they did a pretty good job overall with 2024 and it is much better than I expected it to be.

I would say 5E is by far the best edition of the game, period. I was convinced they were going to break it with 2024 and what was going to come out was going to be terrible or a step backwards. This was reinforced with some of the "duds" they published in UA (Warlock, Wizard, wild shape). But in the end they cleaned that stuff up.

There are things I like better about 2014 (Monk, no Weapon masteries, Indomitable, twin spell), but there are things I like better about 2024 too (origin feats, emanations, exhaustion).

There are some obvious improvements they could have made but didn't make or survey - like simplifying multiclassing and taking away minimums (which I thought was a no-brainer considering BG3), but I still think they did a pretty good job

Mustrum_Ridcully said:

I think the true reason is far simpler - less flawed methodology requires a far more elaborate setup and many more moving parts than they could hope to handle, especially if they have the typical deadlines for "make a new edition".

Seriously, think about how many people you would need to make a representative sample! How do you get those "casuals" to possibly even do an actual playtest? How valuable is feedback based just on reading some rule suggestion actually? If it requries play-test, how long until survey participants have organized a group to play? How many factors are actually influencing the outcome of such a play-test that might be unrelated to the topic of the survey?
How long are they supposed to wait until that feedback is there, and how do they control for all those variables and factors?!
That's almost as complicated at testing new medical drugs! With far lower stakes.

Click to expand...

You create an robot duplicate indistinguishable from an actual teenager and send it out into the wild to invite real teenagers to play D&D and record their responses.

Hey, it works for wildlife!