The Pitfalls of D&D Beyond Data

[FrogReaver]

Who has done this?

Does it matter?

 

[Hawk Diesel]

Any time we get data of any sort which is objective hard data, the same thing happens.

If the data doesn't meet people's personal experience or expectations, they make an argument for why the data isn't perfect and therefore should be dismissed.

Of course, data being imperfect does not mean data should be dismissed. Data can be extraordinarily accurate and very fairly representative, while still being imperfect. Statistics has demonstrated, conclusively, that data can be fairly representative of a whole while remaining imperfect.

We should be able to talk about broad generalizations and trends without constantly being reminded that the data isn't 100% perfect and is not measuring all conceivable games and players and DMs and purchasers and experiences in the world.

I disagree. Because this data is only representative of players that use D&D Beyond. That is a subset of the total population, and thus generalizations or conclusions made based on that data may be skewed based on the qualities shared within that population that are not as well represented in the broader population. To give an extreme example, if a study had thousands of data points, but only included men as the subpopulation, how accurate or useful would the data be for women? But then, WotC may be only caring about that data with D&D Beyond to better target that group rather than the broader population. But somehow I doubt that.

Additionally, as someone else pointed out, I think it would be more helpful to provide the raw data for public examination, since that will allow better discussion while helping to remove bias or blindspots in methodology.

 

[FrogReaver]

I disagree. Because this data is only representative of players that use D&D Beyond. That is a subset of the total population, and thus generalizations or conclusions made based on that data may be skewed based on the qualities shared within that population that are not as well represented in the broader population. To give an extreme example, if a study had thousands of data points, but only included men as the subpopulation, how accurate or useful would the data be for women? But then, WotC may be only caring about that data with D&D Beyond to better target that group rather than the broader population. But somehow I doubt that.

Additionally, as someone else pointed out, I think it would be more helpful to provide the raw data for public examination, since that will allow better discussion while helping to remove bias or blindspots in methodology.

That's always a valid point. The population D&D Beyond is measuring may not actually be all that representative in the first place. But even more importantly IMO, even if it were a representative population, the methodology used to present the graphs they have shown could be so far off that the results could still be wrong/misleading even though the data was measuring a potentially representative population.

 

[Morrus]

we are the ones that have to deal with others using D&D Beyond data as evidence of some point

You keep taking about things you “have” to do. What you mean is things you “want” to do.

 

[FrogReaver]

You keep taking about things you “have” to do. What you mean is things you “want” to do.

I'm willing to have this conversation, are you sure this is really the conversation you want to have? Are you sure it's one you want to have here on a 5e forum? Maybe a philosophy or human nature forum would be more suitable?

 

[MNblockhead]

You keep taking about things you “have” to do. What you mean is things you “want” to do.

Can we cut [MENTION=6795602]FrogReaver[/MENTION] some slack on his overly imperative writing style and just agree that a significant percentage of us geeks enjoy discussing statistics? I find his posts on this and other other thread to be, overall, measured and interesting.

 

[TwoSix]

We don’t need data at all.

Sometimes they share stuff as a conversation point. And it’s fun to talk about. But we don’t need it. It’s just an excuse for us all to flap our lips

I agree that the primary purposes of these charts does appear to be for the lulz.

 

[bedir than]

Does it matter?

If one is offering a responsorial or correction to a stance that doesn't exist their entire effort is vaporware.

Congratulations, you won the debate with.. [checks notes] no one. Not a single person did what you are countering.

 

[Zardnaar]

Agreed, that's definitely a problem if anyone attempts o make a claim about all d&d based on the data. I don't think they are directly guilty of that one, but I think threads on this forum and others often pop up that make such claims (like the 90% of games stop at level 10).

Think the number used to be 95% for AD&D.

This data I think explains a lot about say Pathfinder. Why did people play 3.5 over 4E? Well if you don;'t play high level 3.5 is fine for the most part. Druids have never really been that popular in surveys or even that vibe back in AD&D. They were plenty powerful in 1E and 3.5 but power doesn't seem to tie back to popularity. If you don't play Druids and play at high level a lot of complaints about 3.5 don't exist for the majority of players.

If Druids are 2% of the classes played you only see them once in about 50 characters or about once in 10 games (4-5 person party).

Then 90% of those games don't go to level 10 so even with the most over powered Druid in the world 10% of 1/10 is 1% of games are ruined by Druids. CoDzilla may have been a problem and even in that 1% you need a powergamer. Wizards are 5 times more popular than Druids 2% so a high level wizard is potentially a problem in 5% of games assuming you have a power gamer and then is a wizard any more abusable than any other high level spellcaster.

Sort of gives you an idea I hope of how irrelevant our opinions on forums are to game designers.

 

[FrogReaver]

I think it will be beneficial to work out some solutions for analyzing a dataset of D&D characters. How best to address the multiclassing problem, the subclass problem, the paywall problem and the made vs played problem.

So let's first tackle the paywall problem.

With regular D&D people may test out D&D 5e for free with just the basic 5e pdfs. Those people are definitely part of the population of all D&D players. D&D Beyond may also have some people that are testing out 5e with free content the same way. However, some people (probably a large portion) will already be playing 5e with at least the players handbook and will be trying out D&D Beyond with the free material to see if it's worth spending actual money on for them.

If possible we want to include those players who are trying out D&D there the first time but exclude those players who just make a character from free material to test out D&D Beyond. Those players may also test out leveling a character and some other features as well. I don't think there's going to be a way to separate which players are which. So I would think the best option would be to simply exclude all users that only have access to free content from the comparisons.

In fact I'd go a step further and say the sample population most closely resmembes the actual population of all D&D players are the players that buy the PHB and at least 1 supplemental book on D&D Beyond. Why is that? Because those players are showing they will have access to most of the options they find interesting in the game on D&D Beyond. Home groups tend to already do that by sharing books. So a given player doesn't actually need to own a book like xanathar's to make a hexblade in real life. But on D&D Beyond he would need personally spend money on buying xanathars (at least I don't think there's a group sharing option on D&D Beyond, someone correct me if I'm wrong).


Anyone else have thoughts on this methodology and reasoning?