The Pitfalls of D&D Beyond Data

FrogReaver · Feb 10, 2019

bedir than said:
If one is offering a responsorial or correction to a stance that doesn't exist their entire effort is vaporware.

Congratulations, you won the debate with.. [checks notes] no one. Not a single person did what you are countering.

This phenomenon of people quoting D&D Beyond stats to prove points happens the most right after they post some of their data. Those posts and threads are months old at this point and so I'm not going to dig them up. However, there's been comments about their data in more recent threads, but it's not important enough to go digging through pages and pages of comments looking for the proverbial needle in the haystick that I know is there (because I've recently read it) just to prove you wrong. Instead I'll give you one current example.

The example: @Morrus did. It's still on the front page of this forum and shows in the thread title: "90% of D&D Games Stop By Level 10; Wizards More Popular At Higher Levels"

By the way, why the heck does someone with 762 posts in the last 4 years think he has any grasp on the variety of points and justifications for those points that people actually post on this forum?

Morrus · Feb 10, 2019

FrogReaver said:
By the way, why the heck does someone with 762 posts in the last 4 years think he has any grasp on the variety of points and justifications for those points that people actually post on this forum?

Drop the gatekeeping crap right now, n00b.

FrogReaver · Feb 10, 2019

Morrus said:
Drop the gatekeeping crap right now, n00b.

LOL, I love the n00b insult at the end. It's very entertaining!

ClaytonCross · Feb 11, 2019

FrogReaver said:
LOL, I love the n00b insult at the end. It's very entertaining!

Yes which proves his point in that if post are a measure of validity Morrus's 39,521 post to your 3,940 posts would make you a n00b and irrelevant if we are calling that a valid maker of knowledge base. Which we are not. So while your one of the more interesting posters to read you do have a tendency to get a bit dismissive of posters based on off point arguments like posts. I still came to your thread because you make interesting discussion and I heard you. I have even posted a on the D&D Beyond feature requests asking for player inactive characters tags and a statement on slides listing the day delimiter so as we read slides we have a beater representation of what the slide is depicting. That said, questioning the validity of one someone says based on how much they said instead of the content of their argument makes you look bad, instead of making them look unqualified. If your going to say the statement was in jest... their is no indication in the text of the post that implies that and the person your posting to does not know you well enough to infer that in plain text. Try to not use meta reasons to "win" an agreement through dismissing the debater instead of arguing the point of the debate, please. @Morrus is correct that it is gatekeeping, pointless, and unnecessary for you to be heard, since many of us actually came to your thread to see what you had to say.

Zardnaar · Feb 11, 2019

Eliteism over post counts wow lol.

ClaytonCross · Feb 11, 2019

FrogReaver said:
This phenomenon of people quoting D&D Beyond stats to prove points happens the most right after they post some of their data. Those posts and threads are months old at this point and so I'm not going to dig them up. However, there's been comments about their data in more recent threads, but it's not important enough to go digging through pages and pages of comments looking for the proverbial needle in the haystick that I know is there (because I've recently read it) just to prove you wrong. Instead I'll give you one current example.

The example: @Morrus did. It's still on the front page of this forum and shows in the thread title: "90% of D&D Games Stop By Level 10; Wizards More Popular At Higher Levels"

People also use invalid math to prove points. Example: "Average DPR with % hit", we have talked about it before that this is not an acute representation of actual game play damage being that average DPR is only correct about 3% of the time for any specific battle, does not account for 0 sum damage on miss, or loss of damage from over kill... however, I don't have a better metric for measuring damage. So I use it too sometimes to prove a point as do you. The same is true for D&D Beyond data. As @Morrus pointed out...

Morrus said:
We don’t need data at all.

Sometimes they share stuff as a conversation point. And it’s fun to talk about. But we don’t need it. It’s just an excuse for us all to flap our lips

You are correct that 1. It will be used to argue and 2. it is not accurate... However, when it agrees with an argument anyone is likely to take 3% of proof over 0% proof as a point in their favor and when it disagrees with your argument your likely to say its inaccurate unusable metric that does really support anything without providing any metric to counter it. 3% is more accurate than nothing but a lot less than 100% so both are technically correct. In the end everyone likely to except an answer you already agree with and dismiss one you disagree with.

I find that most conversations are not about changing someone else's opinions but developing your own and finding some level of common understanding which might shift both extremes a little closer. I actually find that to be the bet out come in most discussions. Their are some things that are factually wrong but in an abstract game that exist in imagination their are a lot of if you do this A is true and if you do this B is true.

Example, Hold Person is not a "damage spell" but can be responsible for great deal of damage making it one of the most powerful attack spells in the game... but how do you measure that? People arguing it might bring up some good points of how to use it and work together as a group or even better ways to use direct damage spells. Anyone involved in the debate has potential to be a better player because of the debate and changes to perspective. No is likely to change the mind of people who disagree that hold person does more or less damage than X spell. The conversation becomes the valuable part of the thread not "winning the debate".

Metrics might be useful in this conversation or they might muddy the conversation bogging it down. It really depends on the topic and the context of the discussion. Sometimes 3% is enough to remove the 0 sum arguments just like D&D Beyond data might not be useful reflecting the sum of the D&D community but with a quite large user base they do have enough people and data to argue that a "type of player" exist in quantity vs "its a small percentage of the community" or "no one does that" debate points.

Mistwell · Feb 11, 2019

Blue said:
Actually, I talked about why the data wasn't representative, and then talked about how it matched my own insights. So I can say pretty directly that I wasn't dismissing it because it disagreed with what I thought, I was pointing on the limitations of the data even though it hit my confirmation bias.

So I ask you to move beyond your own assumption that challenging data must do with disagreeing with it - some of us are just rigorous.

Here's what I said:

http://www.enworld.org/forum/showth...opular-At-Higher-Levels&p=7559788#post7559788

I was not talking to you.

Mistwell · Feb 11, 2019

FrogReaver said:
That's because as posters on a message board about 5e we are the ones that have to deal with others using D&D Beyond data as evidence of some point in the context of other discussions when the data they are citing can't possibly be evidence of their point. That makes the discussion become frustrating for all party's involved. Pointing out up front when the D&D Beyond data is presented what it actually shows and doesn't helps alleviate that issue in future discussions.

Nobody has done that.

Prove me wrong.

Mistwell · Feb 11, 2019

Hawk Diesel said:
I disagree. Because this data is only representative of players that use D&D Beyond. That is a subset of the total population, and thus generalizations or conclusions made based on that data may be skewed based on the qualities shared within that population that are not as well represented in the broader population.

Right. This was my statistics argument. It's a MASSIVE subset of the audience. WAY WAY WAY beyond what, for example, polling companies use to draw conclusions. It's not PERFECTLY representative, but it is representative in these number of fair generalizations to the general population of D&D players and DMs.

To give an extreme example, if a study had thousands of data points, but only included men as the subpopulation, how accurate or useful would the data be for women?

Very. If you know what half the population thinks, and you know exactly what demographic it represents, you can easily adjust to come to some generalized conclusions from it.

Additionally, as someone else pointed out, I think it would be more helpful to provide the raw data for public examination, since that will allow better discussion while helping to remove bias or blindspots in methodology.

Except it's not about us. That's their data. They released some general information about it - more than almost any other company does - but you don't have any entitlement to that kind of data and they are not going to reveal their proprietary data.

FrogReaver · Feb 11, 2019

ClaytonCross said:
Yes which proves his point in that if post are a measure of validity Morrus's 39,521 post to your 3,940 posts would make you a n00b and irrelevant if we are calling that a valid maker of knowledge base. Which we are not. So while your one of the more interesting posters to read you do have a tendency to get a bit dismissive of posters based on off point arguments like posts. I still came to your thread because you make interesting discussion and I heard you. I have even posted a on the D&D Beyond feature requests asking for player inactive characters tags and a statement on slides listing the day delimiter so as we read slides we have a beater representation of what the slide is depicting. That said, questioning the validity of one someone says based on how much they said instead of the content of their argument makes you look bad, instead of making them look unqualified. If your going to say the statement was in jest... their is no indication in the text of the post that implies that and the person your posting to does not know you well enough to infer that in plain text. Try to not use meta reasons to "win" an agreement through dismissing the debater instead of arguing the point of the debate, please. @Morrus is correct that it is gatekeeping, pointless, and unnecessary for you to be heard, since many of us actually came to your thread to see what you had to say.

Morrus says that comment was out of line, so for Enworld purposes it is and I won't dispute that. However, since you and @Zardnaar and others may comment on it I do want to add some context to it that I feel is being overlooked.

The intention of that comment was simply to convey the idea that in order to make a claim that something doesn't happen on this forum then you should to be very familiar with this forum. I think questioning someone's familiarity with this forum is a fair game if they start claiming something doesn't happen here. Ultimately though, the way I attempted to convey that idea was over the line. That said, the comment over the line was also not the only comment in that post. In context that post answered the poster with an example like he desired and explained why other examples wouldn't be provided. Only after that did I attempt to raise the question of the poster's familiarity with this forum (with an epic fail of a comment apparently...)

Anyways, that's the full context. Hopefully I didn't cross another line by posting this.

Oh and thanks for the honest evaluation of me! I appreciate it.