D&D General Why DPR Sucks: Discussing Whiteroom Theorycrafting

[NotAYakk]

B. Run repeated Monte Carlo simulations and regression analysis.

That is nonsense.

Monte Carlo simulations are useful because they are fast and easy compared to a full analysis; they have really nice convergence properties.

Stating that they are somehow superior to a full model is frankly mathematically illiterate.

A Monte Carlo simulation is strictly inferior to a full analysis of the same model. It is just much cheaper and often almost as good.

 

[Snarf Zagyg]

That is nonsense.

Monte Carlo simulations are useful because they are fast and easy compared to a full analysis; they have really nice convergence properties.

Stating that they are somehow superior to a full model is frankly mathematically illiterate.

A Monte Carlo simulation is strictly inferior to a full analysis of the same model. It is just much cheaper and often almost as good.

Wait. Hold on. We just heard from one person that they are impossible because, something something chess and it would be impossible to program.

But you're saying not to use them because they are too fast and easy to use!

Well then, since both you and @FrogReaver appear to be much more knowledgeable about math, statistics, and analytics than I am, how about the two of you debate about the best measure to use that isn't DPR and get back to me?

 

[FrogReaver]

Here's the thing, in two parts:

1. You made a claim that "actual data doesn't matter." That's .... the worst thing you can ever say. The data always matters. It might be limited. You might want better data (more data). You might need to be concerned about whether the data is representative for your purposes. You might want the data to better reflect conditions. But actual data always matters. This such a banal point that there are countless jokes that end with the punchline of the clueless guy going, "Yes, that might be how it works in practice, but how does it work in theory?"

I made the claim that because the data we could get wouldn't actually answer the questions we are trying to answer that the actual data doesn't matter in this situation. I never would make a claim that data doesn't matter. It always matters for something.

So please stop trying to twist my position into anit-analyitic/anti-statistical. It's not. My position is that before you waste all this time gathering data to answer a question - you need to be sure the data you are gathering can tell you what you want to know.

2. You don't seem to understand why I am suggesting Monte Carlo simulations and regression analysis, but that's okay! I am always open to alternatives. What is weird (and concerning) is that your objection to it (which is not well founded) is based upon a purported inability to capture nuance ... as compared to DPR? Again, to the extent that looking at probabalistic outcomes is limited (as all things are), it is certainly not nearly as limited by nuance as DPR is.

I do understand why you suggest them. You don't seem to understand why I say they are bad for this. How many different fighter builds are there? You can't monte carlo and design good tactics into the sum for each one over a vast array of situations. And that's just fighters. We have ever other class to do. We have every other combination of 4 classes to do.

Now one might could make a learning AI that optimizes party tactics over an adventuring day toward whatever goals you predefine. That's not monte carlo though. That's machine learning.

And that's the real problem right there; not just that DPR is incredibly limited, but that people that are proponents of it refuse to actually do real math and statistics to come up with better measures. Because that would be ... hard! It's a lot harder to do it right.

What you are suggesting is actually worse than DPR. It doesn't actually more accurately answer the questions we want answered and it is more misleading because people will believe it is more accurate in answering those questions.

Just like it's a lot harder to do advanced analytics and statistics in sports than to say, "That guy went 3 for 5. Therefore, he hit 60% of the balls. I am done now, because anything else would be hard and require nuance."

Either we should do things right, or at a bare minimum stop using DPR as the end-all, be-all metric for comparing unlike things.

DPR is the simplest and most useful metric we have and outside of attempting to use machine learning to tackle the problem it's probably the best single metric we ever will have.

 

[FrogReaver]

Wait. Hold on. We just heard from one person that they are impossible because, something something chess and it would be impossible to program.

But you're saying not to use them because they are too fast and easy to use!

Well then, since both you and @FrogReaver appear to be much more knowledgeable about math, statistics, and analytics than I am, how about the two of you debate about the best measure to use that isn't DPR and get back to me?

Monte Carlo is cheap and fast for a single use case. It's not necessarily cheap or fast when you have a very large number of use cases. It may still be cheaper and faster than a full analysis, but something can be cheaper and faster and still be too computationally heavy to realistically use to answer a question.

 

[Snarf Zagyg]

DPR is the simplest and most useful metric we have and outside of attempting to use machine learning to tackle the problem it's probably the best single metric we ever will have.

No, it's not. But I already made a long post and several follow-up posts regarding why. If you continue to advocate for bad math, nothing will change your mind now. It's just interesting because we've seen this over and over again in different fields, with the "old guard" advocating for a single, terrible statistic, simply because they had facility with it and advanced statistics were harder to compute and more difficult to understand at first.

I really do agree with the people who say DPR (DPS) is a carryover from CRPGs, which is funny.

Anyway, peace out. Enjoy your DPR advocacy with someone else.

 

[Asisreo]

I made the claim that because the data we could get wouldn't actually answer the questions we are trying to answer that the actual data doesn't matter in this situation. I never would make a claim that data doesn't matter. It always matters for something.

The questions we're asking are how effective anything is. And we can get that by analyzing the data.

DPR calculations are just thought-experiments. They're those riddles ted-ex gives where they tell you that something very specific and incongruent is happening and you have to figure out the best course of action given these strange rules.

The same is with DPR.

DPR is the riddle "Assuming all characters are in-range and capable of having any weapon or feat they choose with maximum resources, which of these two scenarios would you choose if you wanted to go purely for damage."

That's all DPR solves, a riddle. Take away the range stipulation, add cover, add terrain, take some resources away, and restrict options. Now, the actual damage on the board is going to be so wildly different than DPR that DPR is obviously useless. And that's not even everything to account for.

So please stop trying to twist my position into anit-analyitic/anti-statistical. It's not. My position is that before you waste all this time gathering data to answer a question - you need to be sure the data you are gathering can tell you what you want to know.

I understand your viewpoint that mindlessly collecting data won't give exact answers, but where else is there to collect the data of how well something performs than in-performance?

Car manufacturers do it. They calculate the ideal environment that they can predict, but cars are often put in unpredictable environments. Terrible weather, crashes, poor terrain. These are all unpredictable, but if not accounted for, can create a car that's amazing in-theory but becomes a deathtrap in real life and would be prone to recalls.

 

[Chaosmancer]

Even in economics, they don't just calculate based on numbers. If they did, we'd probably have economic crashes one-after-another.

Most economists use a model determined using past models. Basically, they look at what happened during a past trend to predict what will happen during a present trend.

No matter what field of study you want to be in, unless it's pure mathematics, you want live data.

Oh, I agree in general terms, I was more talking about the idea of starting a model with "all else being equal". An economic model assumes a lot of things, and essentially holds them static to focus on the effects they want to focus on. Because it is impossible to build a truly accurate economic model that considers every single variable.

Such it is with DnD. We can't possibly talk about every single variable, and DPR is the simplest thing we can talk about, holding all else equal.

I found the thread on Giant in the Playground and it's NEARLY 500 REPLIES LONG. And the only thing obvious from it was he was having to defend himself against a huge group, it went on for a long time, and like most long threads with 20 to 1, it eventually ended with "Well it's OK if you don't like my opinion".

And then you guys came HERE to continue it. In multiple threads. I guess because he stopped responding to you there? Apparently because, in your words, you want his "professional reputation [to] take a damaging blow" as if he's some professional scientist presenting a paper as opposed to a D&D fan who made a video with his opinion about a class sucking?

Not a good look dude.

I know you weren't quoting me, but I am involved in this.

Firstly, Treantmonk came in late and left early on that thread. He might have present for around 100 replies at best. And, since that thread involved people supporting him and criticizing his arguments, it was not really him defending himself against a large group of people.

In fact, I think we have a fairly even split on that thread between supporting him and criticizing him. So, definitely not 20 to 1


But, I think the egregious point here is you acting like we are doing something bad in spreading what he said. If he released a new video tomorrow, recanting some of his points, wouldn't it be relevant to the discussion? Personally, the reason I quoted him over here was because he had explained his methodology better over there, despite his continual arm's length responses. It was over on that thread that I realized how he could possibly claim that every single class was possible of beating his baseline. Because he was counting spells with multi-round durations. This changes the discussion by providing clarity. If you are going to count Animate Objects on ten daggers giving the wizard a bonus action of x10 attacks for 1d4+4 damage... well, no wonder you are saying he can reach the baseline compared to the monk's at-will damage.

I wasn't seeking to criticize that choice (though I do think it is innaccurate for a baseline, especially since it is not what most of us thought he meant) but to bring that clarity he provided over there, over here. And, just like if he had tweeted, or if we had trawled through the youtube comments, Treantmonk's lack of response to criticisms of his paradigms and arguments is a fact that has to be considered.

Just like if I was reading a Bard optimization by Snarf, whom I've noticed hates bards, I think knowing how Treantmonk is approaching the subject of the discussion, when his analysis is what is at the core of the discussion, is important to the topic at hand.

 

[Chaosmancer]

DPR is the riddle "Assuming all characters are in-range and capable of having any weapon or feat they choose with maximum resources, which of these two scenarios would you choose if you wanted to go purely for damage."

That's all DPR solves, a riddle. Take away the range stipulation, add cover, add terrain, take some resources away, and restrict options. Now, the actual damage on the board is going to be so wildly different than DPR that DPR is obviously useless. And that's not even everything to account for.

Yes, exactly. Which is why I made the comparison to Economics.

Just having the fight in difficult terrain changes everything. Just having the fight with easily found cover changes everything.

But how do you calculate how much cover or how often you are dealing with difficult terrain? Do you account for fighting underwater when comparing a spear fighter to a greatsword fighter? The disadvantage is a big thing. But how often are you fighting underwater.


To get accurate answers to those sort of problems, you would need to collect game data from thousands of games. And no one is going to go through that much effort. Or even remember the game in enough detail to give accurate reports of the details you would want to measure.

 

[FrogReaver]

No, it's not. But I already made a long post and several follow-up posts regarding why. If you continue to advocate for bad math, nothing will change your mind now. It's just interesting because we've seen this over and over again in different fields, with the "old guard" advocating for a single, terrible statistic, simply because they had facility with it and advanced statistics were harder to compute and more difficult to understand at first.

I really do agree with the people who say DPR (DPS) is a carryover from CRPGs, which is funny.

Anyway, peace out. Enjoy your DPR advocacy with someone else.

I just love how everything else I said in my long post was ignored...

 

[FrogReaver]

Yes, exactly. Which is why I made the comparison to Economics.

Just having the fight in difficult terrain changes everything. Just having the fight with easily found cover changes everything.

But how do you calculate how much cover or how often you are dealing with difficult terrain? Do you account for fighting underwater when comparing a spear fighter to a greatsword fighter? The disadvantage is a big thing. But how often are you fighting underwater.


To get accurate answers to those sort of problems, you would need to collect game data from thousands of games. And no one is going to go through that much effort. Or even remember the game in enough detail to give accurate reports of the details you would want to measure.

Yep. And even then what matters isn’t the average of the parameters for all d&d games, but rather what the parameters of the games a particular poster plays in are.

D&D is a fairly chaotic system where relatively small changes in parameters can vastly change the statistical outcomes. And even then there’s so much variation in our d20 that any given player can hardly tell random variation in the games they experience from Whether something is actually being better or worse.