D&D 5E (2024) A critical analysis of 2024's revised classes

[FrogReaver]

Likewise if you have no idea how representative a sample is you should not assume that it is not representative either, yet that is exactly what people are doing here. Absence of a reason to confirm the quality of the survey is not the same as "plenty of reasons" to doubt it.

Without any specific information on the test design or metrics, you can only draw conclusions from the data you have. We have no reason to believe the sample is not representative of D&D players. No one who claims with confidence that it is not representative has provided even one specific topic that would have scored differently if the barrier was lower nor presented any evidence that the players who did take the survey (i.e. dedicated players) have different opinions, in aggregate, than players who didn't (i.e. casual players).

We do.

What we know is that in virtually every domain those that invest a lot more time into something generally have different perspectives than those that don’t. So we most definitely can make that inference. It need not be scientifically rigorous for that kind of inference since that’s the general rule and in absence of some specific information that rule doesn’t apply for a specific case.

Moreover, the people commenting here and saying it is not representative, were not the ones that built the survey. Yet we have someone on this thread who was actually part of the team that designed and executed the survey and they are not saying this. The "it is not representative" argument started as a counterpoint to the one person on this thread who actually does know how the survey was designed and posted about the survey results and rules design.

Doesn’t matter. It’s still correct to say a sample isn’t representative when the final results bias toward the non-casual class.

(One might could correct that this, but it’s not clear if they did, 3-5 years of experience feels X but 20+ feels Y, then proceed to weight those classes with your estimated actual people in the population).

I can walk through a city and ask people if they prefer Orange Juice or Apple Juice. If I only poll people inside one city it is possible the results would be different as compared to the population since I did not poll people living in the suburbs or rural areas or in different cities. Just because the results could possibly be different, does not constitute evidence they would be different.

Sure, but if you gate it so that only those that frequent fruit drink cafes are likely to participate then it should be obvious why such a sample isn’t representative even of juice drinkers in that city.

 

[ECMO3]

We do.

What we know is that in virtually every domain those that invest a lot more time into something generally have different perspectives than those that don’t. So we most definitely can make that inference.

Different perspectives is not the same as different preferences, nor does it imply they will have different preferences.

If we know there are differences provide an example - dedicated players prefer A, while casual players prefer B.

It need not be scientifically rigorous for that kind of inference since that’s the general rule and in absence of some specific information that rule doesn’t apply for a specific case.

You some evidence to draw that inference and we have none.

Doesn’t matter. It’s still correct to say a sample isn’t representative when the final results bias toward the non-casual class.

It is not correct to say the results are biased when no evidence of such a bias exists.

We presume that the surveys were taken predominantly by "dedicated" players. There is not even any proof of that, but I am willing to accept it as logical and probably true. That is fundamentally different than assuming this group of "dedicated" people has different preferences than the population that includes casual players. There is no evidence of this at all and it is incorrect to infer the results would be different without such evidence.

We get over and over again that it is "obvious" yet no one can provide a single example of something specific that would be different if more casual players answered the survey.

Sure, but if you gate it so that only those that frequent fruit drink cafes are likely to participate then it should be obvious why such a sample isn’t representative even of juice drinkers in that city.

It is not obvious and you can not make such an inference about the results. If we asked people who frequent fruit drink cafes if they prefer juice or soda you would have a point and we could infer a bias in the answers. Likewise, if we asked people if D&D was a better game than monopoly we could infer some difference in the results as we are polling D&D players. That sort of bias is not what we are talking about here.

There is nothing to suggest skew on the specific game preferences that were asked about based on whether someone is a dedicated player or a casual player.

The people who responded to the survey were people who play D&D and it even asks this in the survey.

 

[Zardnaar]

Different perspectives is not the same as different preferences, nor does it imply they will have different preferences.

If we know there are differences provide an example - dedicated players prefer A, while casual players prefer B.




You some evidence to draw that inference and we have none.




It is not correct to say the results are biased when no evidence of such a bias exists.

We presume that the surveys were taken predominantly by "dedicated" players. There is not even any proof of that, but I am willing to accept it as logical and probably true. That is fundamentally different than assuming this group of "dedicated" people has different preferences than the population that includes casual players. There is no evidence of this at all and it is incorrect to infer the results would be different without such evidence.

We get over and over again that it is "obvious" yet no one can provide a single example of something specific that would be different if more casual players answered the survey.




It is not obvious and you can not make such an inference about the results. If we asked people who frequent fruit drink cafes if they prefer juice or soda you would have a point and we could infer a bias in the answers. Likewise, if we asked people if D&D was a better game than monopoly we could infer some difference in the results as we are polling D&D players. That sort of bias is not what we are talking about here.

There is nothing to suggest skew on the specific game preferences that were asked about based on whether someone is a dedicated player or a casual player.

The people who responded to the survey were people who play D&D and it even asks this in the survey.

This.

Just by posting in forums youre a dedicated player by default almost.

ENworlds not remotely representative lol. We're the most dedicated vocal members of our age group for the most part.

Some small tiny % of 10%ish of the fanbase.

 

[ECMO3]

This.

Just by posting in forums youre a dedicated player by default almost.

ENworlds not remotely representative lol. We're the most dedicated vocal members of our age group for the most part.

Some small tiny % of 10%ish of the fanbase.

Sure and we have wide variances of opinion on the rules of the game. This further illustrates the point that being "dedicated" is not evidence of a bias in preference.

 

[FitzTheRuke]

I have a low opinion of the surveys and don't trust their results are particularly meaningful, but that's because I filled them out and most of the time, I had no idea what I was saying. I don't know if I ever got my opinion across.

If everyone had the same problem (or worse, were sure they were saying one thing, while WotC read it as another) then I don't know how they ever got much useful info out of them.

 

[Zardnaar]

I have a low opinion of the surveys and don't trust their results are particularly meaningful, but that's because I filled them out and most of the time, I had no idea what I was saying. I don't know if I ever got my opinion across.

If everyone had the same problem (or worse, were sure they were saying one thing, while WotC read it as another) then I don't know how they ever got much useful info out of them.

Theater and major trip wires.

 

[FrogReaver]

Different perspectives is not the same as different preferences, nor does it imply they will have different preferences.

If we know there are differences provide an example - dedicated players prefer A, while casual players prefer B.




You some evidence to draw that inference and we have none.




It is not correct to say the results are biased when no evidence of such a bias exists.

We presume that the surveys were taken predominantly by "dedicated" players. There is not even any proof of that, but I am willing to accept it as logical and probably true. That is fundamentally different than assuming this group of "dedicated" people has different preferences than the population that includes casual players. There is no evidence of this at all and it is incorrect to infer the results would be different without such evidence.

We get over and over again that it is "obvious" yet no one can provide a single example of something specific that would be different if more casual players answered the survey.




It is not obvious and you can not make such an inference about the results. If we asked people who frequent fruit drink cafes if they prefer juice or soda you would have a point and we could infer a bias in the answers. Likewise, if we asked people if D&D was a better game than monopoly we could infer some difference in the results as we are polling D&D players. That sort of bias is not what we are talking about here.

There is nothing to suggest skew on the specific game preferences that were asked about based on whether someone is a dedicated player or a casual player.

The people who responded to the survey were people who play D&D and it even asks this in the survey.

I don’t think you understand what a representative survey is. It doesn’t matter that the results might be the same across classes, it matters whether your survey sample mirrored the population proportionally along potentially important class splits. If this isn’t achieved then the survey results are not generalizable, at least without further evidence showing those splits wouldn’t have affected the results.

Example: Using proportional techniques to perform the survey along the class splits.

Note: there are 2 kinds of bias here, response bias and selection bias. You seem to be focused only on response bias and I am focused much more on selection bias.

 

[ECMO3]

I don’t think you understand what a representative survey is. '

I don't think you understand that a biased sample does not by itself mean or indicate the results are skewed.

A biased sample, by itself, provides no evidence at all that the results are skewed.