The Pitfalls of D&D Beyond Data

FrogReaver said:

[MENTION=1]Morrus[/MENTION] has claimed multiple times that the charts represent all characters created and even includes the ones not necessarily played. I'd lean toward him being correct on this but maybe not?

Click to expand...

Morrus said:

It's what they said in the video. I'm just the messenger.

Click to expand...

Mistwell said:

Any time we get data of any sort which is objective hard data, the same thing happens.

If the data doesn't meet people's personal experience or expectations, they make an argument for why the data isn't perfect and therefore should be dismissed.

Of course, data being imperfect does not mean data should be dismissed. Data can be extraordinarily accurate and very fairly representative, while still being imperfect. Statistics has demonstrated, conclusively, that data can be fairly representative of a whole while remaining imperfect.

We should be able to talk about broad generalizations and trends without constantly being reminded that the data isn't 100% perfect and is not measuring all conceivable games and players and DMs and purchasers and experiences in the world.

Click to expand...

FrogReaver said:

Your right we don’t need data that’s 100%.

We do need the actual data analyzed with sound methodology or those broad generalizations you talk about won’t have any value.

the methodology used to create these graphs was so bad that they are essentially meaningless.

Click to expand...

Mistwell said:

Any time we get data of any sort which is objective hard data, the same thing happens.

If the data doesn't meet people's personal experience or expectations, they make an argument for why the data isn't perfect and therefore should be dismissed.

Click to expand...

Blue said:

I'd put two caveats on this data.

First, there's no separation between played characters and try-a-build characters, so we don't know this is the proper breakdown for campaigns. I wonder if they can remove characters that have never been given XP.

Second, this is rather self-selecting, for those who use DDBeyond.

For that, it's still an interesting insight that matches my own observations, but that doesn't mean we shouldn't be aware of possible weaknesses in the data.

Click to expand...

Morrus said:

We don’t need data at all.

Sometimes they share stuff as a conversation point. And it’s fun to talk about. But we don’t need it. It’s just an excuse for us all to flap our lips

Click to expand...

FrogReaver said:

The flip side is that if we don't have a discussion about the data and it's pitfalls then many posters draw mistaken conclusions about the data they are showing.

Click to expand...

Blue said:

Actually, I talked about why the data wasn't representative, and then talked about how it matched my own insights. So I can say pretty directly that I wasn't dismissing it because it disagreed with what I thought, I was pointing on the limitations of the data even though it hit my confirmation bias.

So I ask you to move beyond your own assumption that challenging data must do with disagreeing with it - some of us are just rigorous.

Here's what I said:

http://www.enworld.org/forum/showth...opular-At-Higher-Levels&p=7559788#post7559788

Click to expand...

Morrus said:

And that's bad because...? Somebody's wrong on the internet?

Click to expand...

FrogReaver said:

That's because as posters on a message board about 5e we are the ones that have to deal with others using D&D Beyond data as evidence of some point in the context of other discussions when the data they are citing can't possibly be evidence of their point. That makes the discussion become frustrating for all party's involved. Pointing out up front when the D&D Beyond data is presented what it actually shows and doesn't helps alleviate that issue in future discussions.

Click to expand...