D&D General Why DPR Sucks: Discussing Whiteroom Theorycrafting

[Snarf Zagyg]

Statistics and data can’t answer the types of questions we are asking. There’s no way for the data to differentiate the character from the player. As such the best data can tell us is how much a class contributes with an average player using it. The case could very much be the opposite with expert players or poor players. Then there’s the DM, which matters even for published adventures. Then there’s published adventures which many don’t actually play.

All your data would be able to tell me at the end of the day would be how average players playing a class with a specific build perform in the average dms run through of an average published adventure. Thats not a question anyone cares about.

Recognizing the limitations of data is the most important part of analytics.

It depends on the amount of the data. The assumptions that you are using are likely to be incorrect, given a large enough sample size.

For example, if you are using just the data set of Critical Role, then it will necessarily be skewed because of the difference between that game and other games.

On the other hand, if your data set was the theoretical "all games of 5e, ever," then it would include expert players, poor players, and average players. Arguing against the data set would be similar to the baseball player who says keep complaining about the one time he was "robbed" of a home run in his career, forgetting that over the long term (4-5 at bats per game, 162 games per year, 15 year career) the sample size of "robbed" vs. "lucky carry over the wall" likely balances out.

Or, as I keep saying, there are at least two other good ways to do the math:

A. Use actual data; or
B. Run repeated Monte Carlo simulations and regression analysis.

DPR is a very basic tool, and isn't really "analytics." It's just a more advanced version of saying, "Should I use a dagger (2.5) or a short sword (3.5)?" It's fine to resolve a binary choice, but it's explanatory power doesn't really work past that.

 

[FrogReaver]

That's why I suggested having an actual play-by-post log, because we'll have a record on exactly what happened.

It wouldn't matter if the player was good or bad, the data will speak volumes. If a player was playing suboptimally, we'd see that. If the DM decided to homebrew a monster, we'd see the hombrew monster's stats.

And we can use that data to determine how often a player will encounter specific sets of monsters or the difficulty of playing a class. These are questions we brush aside because DPR is seen as more important.

No more assuming the average monster a party will face at a specific level is a specific CR, we'll have the data. No more assuming mobility does or does not play a large role in combat, we'll have the data. No more assuming most monsters in a campaign have resistance or immunity to nonmagical attacks, we'll have data.

And we can further organize data into subcategories. Out of all the DM's that use homebrew monsters, which types are more prominent? How often does a DM that allows short rests add time-pressures?

We'll have a plethora of answers that people throw away to make assumptions. But honestly, more often than not, data usually doesn't show what's expected.

The data you are getting won’t tell you any of that.

Take a play by play of a football game. You get a summary of the actions taken. The running back may have gained 30 yards. Was it due to a great o line. A great running back. The wrong defense called. You can’t answer those questions no matter how many play by play stats you have.

Nor could a person go through and evaluate that individually for the hundreds of thousands of data points you would have.

The data, even if you had it will never be able to tell you what you want it to.

 

[Asisreo]

The data you are getting won’t tell you any of that.

Take a play by play of a football game. You get a summary of the actions taken. The running back may have gained 30 yards. Was it due to a great o line. A great running back. The wrong defense called. You can’t answer those questions no matter how many play by play stats you have.

Nor could a person go through and evaluate that individually for the hundreds of thousands of data points you would have.

The data, even if you had it will never be able to tell you what you want it to.

When I say "play-by-post," I mean every action is a new picture.

A football game captured on TV is a collection of pictures. Each picture moves the action along. With the collection of pictures, you can have it sped up so fast that it imitates live-action. However, what you can't do with live action is stop it, rewind it, or fast forward it. Blink, and you miss it. But with a football game on TV, we can stop it and analyze every bit of it or slow it down. We can see whether it was the O-line or quarterback or running back that pulled their weight in a play because we have the pictures.

Likewise, each action the PC's or DM makes is another picture that moves the action along. If we have it unrecorded, you won't be able to slow it down or stop it. However, if it's recorded, it's just like a football game on TV. We can stop a play, rewind it, look at the results, and the experts can analyze it to see what went right or wrong. They can tell you whether the tight-end made the screen work or not.

We've organized the data, but if anyone wants to go back into a specific time where Brady ran 40 yds for the touchdown or when Eltrian casted haste on the barbarian, the actual logs are there, and we can actually analyze them.

 

[Snarf Zagyg]

The data you are getting won’t tell you any of that.

Take a play by play of a football game. You get a summary of the actions taken. The running back may have gained 30 yards. Was it due to a great o line. A great running back. The wrong defense called. You can’t answer those questions no matter how many play by play stats you have.

Nor could a person go through and evaluate that individually for the hundreds of thousands of data points you would have.

The data, even if you had it will never be able to tell you what you want it to.

That's a terrible example, because that's what is being done right now in football. Yards after contact (for example) is a basic stat, and you can easily find (for example) DYAR.

There are two issues: data collection (what data you have) and data extrapolation. I don't think you're familiar with advanced stats in football, but we know a great deal more than just the RB 30 yds.

 

[seebs]

I have seen a lot of analysis of combat here that struck me as good analysis for a spherical melee of uniform density, but totally irrelevant to actual play.

 

[FrogReaver]

It depends on the amount of the data. The assumptions that you are using are likely to be incorrect, given a large enough sample size.

For example, if you are using just the data set of Critical Role, then it will necessarily be skewed because of the difference between that game and other games.

On the other hand, if your data set was the theoretical "all games of 5e, ever," then it would include expert players, poor players, and average players. Arguing against the data set would be similar to the baseball player who says keep complaining about the one time he was "robbed" of a home run in his career, forgetting that over the long term (4-5 at bats per game, 162 games per year, 15 year career) the sample size of "robbed" vs. "lucky carry over the wall" likely balances out.

Or, as I have repeated now twice, there are two good ways to do the math:

A. Using actual data; or
B. Running repeated Monte Carlo simulations and using regression analysis.

DPR is a very basic tool, and isn't really "analytics." It's just a more advanced version of saying, "Should I use a dagger (2.5) or a short sword (3.5)?" It's fine to resolve a binary choice, but it's explanatory power doesn't really work past that.

Both of those will not actually work.

Monte Carlo is terrible because d&d is more complex than chess and there’s no way you’ll ever program in the required nuance for a variety of situations.

Actual data doesn’t do any better. There’s too many dependencies within it, or potential ones at least.

 

[Snarf Zagyg]

Both of those will not actually work.

Monte Carlo is terrible because d&d is more complex than chess and there’s no way you’ll ever program in the required nuance for a variety of situations.

Actual data doesn’t do any better. There’s too many dependencies within it, or potential ones at least.

Um .....

I appreciate your input, but this makes it clear that you seem to lack a certain foundation in analytics.

 

[FrogReaver]

Um .....

I appreciate your input, but this makes it clear that you seem to lack a certain foundation in analytics.

Understanding what data can actually tell you is the foundation. The data in this situation cannot tell you what you think it can.

 

[Snarf Zagyg]

Understanding what data can actually tell you is the foundation. The data in this situation cannot tell you what you think it can.

Here's the thing, in two parts:

1. You made a claim that "actual data doesn't matter." That's .... the worst thing you can ever say. The data always matters. It might be limited. You might want better data (more data). You might need to be concerned about whether the data is representative for your purposes. You might want the data to better reflect conditions. But actual data always matters. This such a banal point that there are countless jokes that end with the punchline of the clueless guy going, "Yes, that might be how it works in practice, but how does it work in theory?"

2. You don't seem to understand why I am suggesting Monte Carlo simulations and regression analysis, but that's okay! I am always open to alternatives. What is weird (and concerning) is that your objection to it (which is not well founded) is based upon a purported inability to capture nuance ... as compared to DPR? Again, to the extent that looking at probabalistic outcomes is limited (as all things are), it is certainly not nearly as limited by nuance as DPR is.

And that's the real problem right there; not just that DPR is incredibly limited, but that people that are proponents of it refuse to actually do real math and statistics to come up with better measures. Because that would be ... hard! It's a lot harder to do it right.

Just like it's a lot harder to do advanced analytics and statistics in sports than to say, "That guy went 3 for 5. Therefore, he hit 60% of the balls. I am done now, because anything else would be hard and require nuance."

Either we should do things right, or at a bare minimum stop using DPR as the end-all, be-all metric for comparing unlike things.

 

[Paul Farquhar]

Understanding what data can actually tell you is the foundation. The data in this situation cannot tell you what you think it can.

What data can tell you is: nothing.

It's a game, go away, have fun, stop wasting time.