The Pitfalls of D&D Beyond Data

[ClaytonCross]

The subclass chart was the biggest offender. It's headers didn't accurately define what it was showing. It was showing the breakdown of active characters given that they had a subclass. The circle graph header defined it as showing the breakdown of subclasses of all active characters. So whether or not he declared it accurate it wasn't.

I could be wrong but that sounds like a problem with clear headers stating what data is presented not the value or accuracy of the data. I understand your point but when I tried to make a simple header to disclaimer that I realized that having to state classes without sufficient levels used to include subclasses or in which subclass were not selected is redundant to "SUBCLASS DISTRIBITION (Active Characters)" since it could be implied that lack of Distribution by due to insufficient levels to be eligible for subclass election or because the player has opted not to make a selection would fail to qualify for the survey. I understand you complaints about multi-classing but in that case NOT counting distribution of subclasses by characters who have a subclass from another class would be less accurate since poll amounts to the number of a subclass by someone who elected a class. Evan as a multi-class both classes and both subclasses were elected and eligible. Not reflecting that would make data incorrect with stated goals.

That does mean some characters are counted twice if they have a second class with a subclass. But it also means no class and subclass was selected on an active character that was not polled. Not polling those subclasses would make the pole entirely wrong for sake of poling which subclasses were selected each time a class allowed a subclass selection and one was chosen.

 

[ClaytonCross]

That could be a disclaimer attached to most of the threads here. Lots of interesting discussions, but too little recognition (in my opinion) that we are the outliers in this hobby. And maybe also too little recognition that our fanaticism, and our tenure*, do not give our opinions and preferences any elevated importance.

*I always laugh when somebody "casually" tries to work into their arguments how long they've been playing D&D, as if that gives them some kind of authority.

The new Forum Creed
"Polls and actions that include players that are not at my table don't represent my table and my table does not reflect the actions or respect the polls of other tables. As my table is about 6 out of 9 million or whatever, so as a rule we are an outlying minority unless we just happen to agree. Since we are the only players I deal with our experiences at our table establish my belief in what normal play is despite being a tiny minority because our normal is my normal. Not everyone else's."

How is that, lol. ; )-

 

[doctorbadwolf]

I

So I think the multiclassing problem has been solved. There’s at least 2 solid solutions for dealing with it.

Any ideas on the best way to handle the subclass problem?

Bad eye already solved it, before posting the graph. He explained it to you in the other thread.

 

[FrogReaver]

I

Bad eye already solved it, before posting the graph. He explained it to you in the other thread.

No he didn’t.

 

[FrogReaver]

I could be wrong but that sounds like a problem with clear headers stating what data is presented not the value or accuracy of the data.

The data I’m referring to is the circle graphs they provided us. If the title is wrong, then The title being wrong makes that circle graph, that I refer to as data, inaccurate. If you mean the dataset that is based on, I don’t claim that is inaccurate. If you mean what they finally stated the subclass chart was actually showing, I don’t think that is inaccurate compared to their claim of what it shows either. I do think they can claim it represents something that is different than what the title claims it represents.

The below is bit of a side point to the above point but,

Even after their clarification I think the methodology used only means the graph presented basically just shows that clerics are a free class, that life cleric is the only free subclass for cleric and that clerics get there subclass at level 1. That’s what the subclass graph seems to be showing IMO. Or maybe I should more accurately say it’s plausible that’s all the graph is showing and so that highly devalues the graph even as an interesting factoid in a conversation about player preferences.

Im replying on my phone and it’s difficult to split quotes up as needed. I’ll add more later

 

[Umbran]

Apology accepted.

The snark does not improve the conversation. Take it elsewhere next time.

 

[ClaytonCross]

The data I’m referring to is the circle graphs they provided us. If the title is wrong, then The title being wrong makes that circle graph, that I refer to as data, inaccurate.

The title of the circle graph for subclass is "SUBCLASS DISTRIBITION (Active Characters)" based on how they answered your questions... that is not a inaccurate title. It is exactly what they said it is. Before having their answers I agreed that they could have been more specific and they could still have put a time delimiter but, seeing the "how and why", I now realize I was wrong about the title. It is what they say it its. I think you are doing the same thing I was doing. We both wanted a slide that indicated "favorite active subclass in the last 120days" but I realize that is not what this data is and the reason it SEEMS inaccurate is because we wanted something more tangible than it is. Its a top layer over view just like BadEye said. The subtext your asking for is not their because the issues you raised prevent them from going any deeper and getting the useful data your looking for/expecting.

If you mean the dataset that is based on, I don’t claim that is inaccurate. If you mean what they finally stated the subclass chart was actually showing, I don’t think that is inaccurate compared to their claim of what it shows either. I do think they can claim it represents something that is different than what the title claims it represents.

So this is exactly my point the bold above is in exact contrast to the underlined part based on subtext of what "they claim it represents" but that's not the disconnect here because they claimed it is exactly as stated. A high level subclass distribution count of active characters with subclasses. This is where you diverge from them. You want something more than that and added a subtext statement in your mind that this slide carried more meaning that it does. I don't blame you, because I did the exact same thing. Until I realized that, I was on board with a title change to be more accurate. What made me realize my mistake was when I started my last post I initially planned to purpose an improved title to support your claim but as I looked at the data they were offering, how they got it, and what was useful for …. I realized my attempts to improve it became less accurate as a reflection of what I wanted not what they were providing. It was at that point I realized... their title is actually spot on. My perception of what they were showing based on what I wanted to see, was off.

So as an experiment / challenge, trying doing what I tried and see if you can do better. Come up with a title that better represents the claim of "what it represents" and see if it then aligns with what they said about being a high level over view or if the new title is want you want instead of what they said the were showing you.

The below is bit of a side point to the above point but,

Even after their clarification I think the methodology used only means the graph presented basically just shows that clerics are a free class, that life cleric is the only free subclass for cleric and that clerics get there subclass at level 1. That’s what the subclass graph seems to be showing IMO. Or maybe I should more accurately say it’s plausible that’s all the graph is showing and so that highly devalues the graph even as an interesting factoid in a conversation about player preferences.

Im replying on my phone and it’s difficult to split quotes up as needed. I’ll add more later

So your proving my point here. Your asking for accuracy in showing player preference but they never stated that as a goal. I made the same mistake. All this shows is subclass distribution among active classes that are eligible and have selected subclasses. You are correct that this metric is not valid for determining player preferences because access to additional options might expand and alter player preference choices. If players only have one Cleric subclass to choose and they pick it then the only choice that represents is a choice of cleric not in the only cleric subclass available to them because there is no choice without a second option. However, we were wrong about that ever being the intent of the data. The title was DISTRIBITION not preference. They said exactly what they measured. Just not what we wanted or expected from a list of how many people picked a subclass because we wanted it as a measure of what people like. They new what they actually had, and carefully worded the title to depict exactly that. So I understand the expectation of what the chart could represent however I was taking it as does represent when that is simply not the case. I think that shows awareness on their part that there is not a good way to show preference when not everyone has access to all classes or if players are just testing ideas to see what they like but don't actually like it enough to use it. This does mean the data is less useful than some might assume up front. It is however not inaccurate. It could still be delimited with poll range of X days but that would not make it what we were asking it to be. For example BadEye said even with those who have access to more material the Life Cleric is very popular so with access to the raw data they can go further than the single graph allows. We can't. They would have to delimit the poll area more for that. "SUBCLASS DISTRIBITION (Active Characters) used by players with access to all subclass who did not multi-class with it" would be a different much much smaller poll that would represent more of what your looking for but that is not what they wanted to show, just want you wanted to see.

 

[FrogReaver]

I understand your point but when I tried to make a simple header to disclaimer that I realized that having to state classes without sufficient levels used to include subclasses or in which subclass were not selected is redundant to "SUBCLASS DISTRIBITION (Active Characters)" since it could be implied that lack of Distribution by due to insufficient levels to be eligible for subclass election or because the player has opted not to make a selection would fail to qualify for the survey.

A few thoughts.

1. "Subclass Distribution (Active Characters)" = "The distribution of subclasses of active characters". "No subclass" is a valid part of the "distribution of subclasses of active characters" and for such a title it needed to be included in the graph for the title to actually match the graph. So I really don't buy your claim that the title of the graph was ambiguous about whether all active characters or just those with subclasses were included in the graph. It was clear IMO.

2. The "Subclass Distribution (Active Characters)" graph was not shown in isolation. It was shown alongside a "Class Distribution (Active Characters)" graph. When 2 graphs are shown back to back with the same (Active Characters) or (whatever) designation that's supposed to mean that they are showing different breakdowns of the same population. As we now know, that's not the case for these 2 graphs.

3. A better name would have been "Subclass Distribution (Active Characters with a subclass)". If that was deemed to long for the title then near the bottom of the graphic I would have at least included a fine print disclaimer about only classes with a subclass being included.

4. There are various subclass distributions they could have shown. I think they chose the least useful distribution to show. However, you are correct that this point doesn't make the graph inaccurate of misleading. That part of what you are trying to say I agree with.

I understand you complaints about multi-classing but in that case NOT counting distribution of subclasses by characters who have a subclass from another class would be less accurate since poll amounts to the number of a subclass by someone who elected a class. Evan as a multi-class both classes and both subclasses were elected and eligible. Not reflecting that would make data incorrect with stated goals.

That does mean some characters are counted twice if they have a second class with a subclass. But it also means no class and subclass was selected on an active character that was not polled. Not polling those subclasses would make the pole entirely wrong for sake of poling which subclasses were selected each time a class allowed a subclass selection and one was chosen.

I've tried to understand this part most of the day and I don't really understand what you are saying enough to make a productive comment. Maybe you can dumb it down for me?

 

[FrogReaver]

The title of the circle graph for subclass is "SUBCLASS DISTRIBITION (Active Characters)" based on how they answered your questions... that is not a inaccurate title. It is exactly what they said it is.

I disagree. I think my previous post provided the reasons why and I don't want to belabor the disagreement here.

Before having their answers I agreed that they could have been more specific and they could still have put a time delimiter but, seeing the "how and why", I now realize I was wrong about the title. It is what they say it its.

I am glad you can change your mind. I can to as long as I see a sufficient reason to do so.

I think you are doing the same thing I was doing. We both wanted a slide that indicated "favorite active subclass in the last 120days" but I realize that is not what this data is and the reason it SEEMS inaccurate is because we wanted something more tangible than it is. Its a top layer over view just like BadEye said. The subtext your asking for is not their because the issues you raised prevent them from going any deeper and getting the useful data your looking for/expecting.

Maybe I am, maybe I'm not but please don't turn this discussion into an attempt to prove I am doing the same thing you were.

I agree that as long as the title matches the graph that the graph is accurate. Though as I think you seem to indicate, an accurate graph can still be a very inaccurate way to measure player preferences for D&D as a whole BUT they aren't trying to measure player preferences as a whole (even though that's what a good portion of our community tends to use such data for), they are just quantifying player preferences on D&D beyond and for that purpose the graph can be 100% accurate (barring the title snafu of course).

So part of what I'm attempting to say on this thread is that for what us forum goers are generally interested in, D&D Beyond data they provided us is a very inaccurate measure of D&D player preferences.

So this is exactly my point the bold above is in exact contrast to the underlined part based on subtext of what "they claim it represents" but that's not the disconnect here because they claimed it is exactly as stated. A high level subclass distribution count of active characters with subclasses.

I disagree and I've stated my case for why already. I don't think we need to keep repeating ourselves on that part.

This is where you diverge from them. You want something more than that and added a subtext statement in your mind that this slide carried more meaning that it does. I don't blame you, because I did the exact same thing. Until I realized that, I was on board with a title change to be more accurate. What made me realize my mistake was when I started my last post I initially planned to purpose an improved title to support your claim but as I looked at the data they were offering, how they got it, and what was useful for …. I realized my attempts to improve it became less accurate as a reflection of what I wanted not what they were providing. It was at that point I realized... their title is actually spot on. My perception of what they were showing based on what I wanted to see, was off.

1. I want the title and presentation and graphics to all match. I believe it doesn't. As long as I believe that then it's fair for me to describe that situation as inaccurate right?
2. I agree with you that what the graph they gave isn't what we as a community wanted. It's useful though, it's just not very useful for us IMO.

So as an experiment / challenge, trying doing what I tried and see if you can do better. Come up with a title that better represents the claim of "what it represents" and see if it then aligns with what they said about being a high level over view or if the new title is want you want instead of what they said the were showing you.

My favorite is "Subclass Distribution (Active Classes with a Subclass)"

 

[FrogReaver]

So your proving my point here. Your asking for accuracy in showing player preference but they never stated that as a goal.

I agree with you that showing D&D player preferences wasn't a primary goal. Showing D&D beyond player preferences was the goal. That doesn't mean I can't talk about how accurate the graphs they provided us are for showing D&D player preferences. Just because that's not their goal doesn't mean people in the community won't attempt to use their graphs for that goal. I think it's good to have an answer about whether those graphs are useful for that goal and why they are or aren't.

I made the same mistake. All this shows is subclass distribution among active classes that are eligible and have selected subclasses. You are correct that this metric is not valid for determining player preferences because access to additional options might expand and alter player preference choices. If players only have one Cleric subclass to choose and they pick it then the only choice that represents is a choice of cleric not in the only cleric subclass available to them because there is no choice without a second option.

Okay, it's good to know we aren't disagreeing about that point.

However, we were wrong about that ever being the intent of the data. The title was DISTRIBITION not preference. They said exactly what they measured. Just not what we wanted or expected from a list of how many people picked a subclass because we wanted it as a measure of what people like. They new what they actually had, and carefully worded the title to depict exactly that. So I understand the expectation of what the chart could represent however I was taking it as does represent when that is simply not the case. I think that shows awareness on their part that there is not a good way to show preference when not everyone has access to all classes or if players are just testing ideas to see what they like but don't actually like it enough to use it.

I think it can be both "showing player preference wasn't their intent" and "they still inaccurately portrayed their graph". That said it's good to note that wasn't their intent and I agree it also shows that they respect and understand what they data shows and doesn't show.

That said my biggest concern is not with their intent but rather how our community will use the data provided. I fear most are going to make the same mistake you did at first. Heck, I'll even admit I initially took the data intent that way, however I don't think any of my claims actually depend on the intent of the data and that's why I'm not backing away from them now.