The Pitfalls of D&D Beyond Data

[Dausuul]

And that's bad because...? Somebody's wrong on the internet?

If not for the imperative to correct people who are wrong on the internet, ENWorld would be a ghost town.

 

[FrogReaver]

Nobody has done that.

Prove me wrong.

I can't I'm too n00b to do that...

JK, For starters there is a thread titled "90% of D&D Games Stop By Level 10; Wizards More Popular At Higher Levels" on the front page of this forum.

 

[Hawk Diesel]

Right. This was my statistics argument. It's a MASSIVE subset of the audience. WAY WAY WAY beyond what, for example, polling companies use to draw conclusions. It's not PERFECTLY representative, but it is representative in these number of fair generalizations to the general population of D&D players and DMs.

Sure, not disputing the value of the data. But it also depends on the questions you are asking and what purpose the data will be used for. If WotC is trying to use this data to determine how they approach future content, are they appealing more to the audience that already buys and uses their D&D Beyond? Or are they trying to continue to broaden their audience and get more people to use D&D Beyond? This data is far more valuable for one of those questions than the other.

Very. If you know what half the population thinks, and you know exactly what demographic it represents, you can easily adjust to come to some generalized conclusions from it.

Only to a point. Once again, it depends on how the data is analyzed, what questions are being asked, and how those conclusions will be used and later assessed. If the question being asked is, "How do we get more ladies playing D&D?" and you only have dudes responding, then you aren't likely to get much value from the data or its conclusions.

Except it's not about us. That's their data. They released some general information about it - more than almost any other company does - but you don't have any entitlement to that kind of data and they are not going to reveal their proprietary data.

Exactly right. It's not about us. I never claimed that WotC had any duty to release its raw data, especially because it may be proprietary (just that doing so would allow for improved ability to reduce bias and blind spots). But when these things are released, it is important to ask why. This is not done in a vacuum, and while the data and conclusions are cool, a company that is built to make a product and money is not going to release something without a strategy behind it. And so we gotta ask, what is WotC trying to do by releasing this data? How is this going to help them drive sales or increase their relevance as a company? Because we as consumers have no ability to challenge these conclusions without the RAW data, and we cannot attempt to understand their own bias in how they see or analyze the data. WotC is not a charity or a scientific organization. But they are trying to make money. And so, not only are these graphs and their conclusions probably just a commercial for D&D Beyond (Hey shareholders! Look at all this data we can use and how it can shape our future products!), but they can also put out literally any numbers or conclusions they want since no one outside the company has access to that raw data.

 

[Mistwell]

I can't I'm too n00b to do that...

JK, For starters there is a thread titled "90% of D&D Games Stop By Level 10; Wizards More Popular At Higher Levels" on the front page of this forum.

That's just a title of an article though. Show me people (you said people, as in plural) were saying it. They are not.

 

[Mistwell]

Sure, not disputing the value of the data.

Frog is. That's what my response is about. You appear to disagree with him. He disputes the value of the data.

 

[Hawk Diesel]

I think it's fair to question the data we have been given and the conclusions WotC has made. But as I said before, it makes me wonder why this is being released. What is WotC's goal in providing these graphs?

But I don't think the data is without value. But then, we really haven't been given data. We've been given suggestions of data filtered through WotC. And I'm not sure what we are expected or even can do with it.

I guess that's why I'm even engaging in this discussion. It just seems so bizarre to me. Like if Coke made a press release providing graphs of how many people drink various Coke products. I mean, obviously they know their sales figures, but what good does it do for the general public that people buy more Coke than Cherry Coke? How do we know if they are including their fountain drink distributors, and how those are calculated in? How would such information grow their business or change how people perceive it? Like I said... bizarre.

 

[Azzy]

I think it's fair to question the data we have been given and the conclusions WotC has made. But as I said before, it makes me wonder why this is being released. What is WotC's goal in providing these graphs?

It's not WotC that's providing this—it's the D&DBeyond team. This is data that they collect (and share with WotC) to better understand their consumer base. They probably also think that the data is interesting to the D&DBeyond community (and maybe the D&D community in general).

Personally, I do find it interesting as it tells us about a section of the larger D&D community. Can we extrapolate from this sample to the larger population? Not without error (to say the least), to be sure, but it does allow us to see part of the bigger picture outside our narrow experiences.

 

[Zardnaar]

Note the numbers involved with this data are more than political polling.

It was mostly kind of assumed that games were mostly low level and the basics were the most popular.

Even in AD&D Fighters were way more popular than Druids.

The only mild surprise was Tieflings and Dragonborn being as popular as they were. Even then it wasn't a massive surprise as power and popularity are not related (3.5 fighters for example).

 

[FrogReaver]

Frog is. That's what my response is about. You appear to disagree with him. He disputes the value of the data.

I’m disputing the value of misleading and inaccurate data.

 

[5ekyu]

It's not WotC that's providing this—it's the D&DBeyond team. This is data that they collect (and share with WotC) to better understand their consumer base. They probably also think that the data is interesting to the D&DBeyond community (and maybe the D&D community in general).

Personally, I do find it interesting as it tells us about a section of the larger D&D community. Can we extrapolate from this sample to the larger population? Not without error (to say the least), to be sure, but it does allow us to see part of the bigger picture outside our narrow experiences.

This is a key element... the DDB releases have been prefaced several times by saying they get commrnts feedback asking for these.

They are an easy visual that sparks interest and discussion.


Anybody thinking WotC is taking **these** graphs to make decisions as opposed to more detailed data dives is imo ofg.