D&D 4E Showing the Math: Proving that 4e’s Skill Challenge system is broken (math heavy)

[Tervin]

Yes, that was the cumulative. What we want is the survival function 1-P(X). See my previous post.

I am sorry, but your result is still wrong. I know this sounds cocky, but I know my method (which takes longer to do, but uses math that I can get most high school students to do) works, and produces the correct answer.

To try and explain in other words than the OP, hopefully making it easier to follow:
We want to know the probability of getting 4 successes before we get 2 failures. This can only happen within 5 attempts, as we will always have reached 2 failures or 4 successes by then.
To keep the math easy to understand I will calculate the probability of the two possible successful cases, and then add them together.

Case 1:
We get 4 successes in a row, wohoo!
0.55^4=0.0915

Case 2:
We get a failure in 1 of our 4 first rolls, but luckily succeed using our 5th roll.
0.55^4*0.45*4
I guess I should stop and explain this one... Out of 5 rolls 4 are successful. Those four rolls are described just as in Case 1. Then we take into account that one roll is a failure, which means we have to multiply the result by 0.45. Finally all we know as that 1 of the first 4 rolls failed. To take into account that there are 4 such cases, we multiply the result by 4.

Total result is 0.55^4+0.55^4*0.45*4=0.256125

Edit: I see that Eldorian, who is clearly my superior when it comes to math, reached exactly the same answer as I did.

 

[Eldorian]

I'd crown myself king of the thread, but that's the OP's title, since he was right. Anyways, for those of you trying to follow this (and because I'm bored and/or gearing up for teaching in 5 hours), I'll explain the differences in the two methods in a simple way!

The OP does the following: he counts

SSSSS

SSSSF
SSSFS
SSFSS
SFSSS
FSSSS

And assigns a probability to each of these 6 outcomes.

The "negative binomial random variable" on the other hand, assigns probability to each of these 5 outcomes.

SSSS

FSSSS
SFSSS
SSFSS
SSSFS

The second one requires the ideas from the first one to do, and the OP basically derived the formula for the negative binomial basically by accident, in the process of solving this problem, so I give him kudos for a direct proof. In fact, he didn't even use the words binomial random variable (which is what I thought of). He went from the base principals of counting and the multiplication principal for independent events. If you were learning this stuff in a class (the one I taught last semester actually) you would have been able to follow the OP's method a chapter before the others.

 

[Tav_Behemoth]

I'm digging the statistical analysis, but here's some stuff I'm able to contribute:

- The table of "Skill Check Difficulty Class" on DMG p. 61 repeats the table of DCs from p. 42, but adds 5 because this is explicitly for skills. That supports the footnote on p. 42 in showing that the base DCs are for ability checks, and they're supposed to be increased when using them for skills.

- The level of the skill check determines the DC of the challenge, right? But the DCs run in bands of three. A level 1 skill check and a level 3 skill check use the same DCs, so the only difference between them is that one of them awards more experience points.

Re: playtester's experience: during the slow patch of a session I made a die out of blu-tak. I then got interested in determining how it was skewed, and started tallying its rolls. Rolling a six-sided was fast, and I had statistical analysis tools on hand (thanks to the chi square table being helpfully presented in Paizo's Dragon Compendium). Nevertheless, even though it was easy to rack up observations in a row, at the end of the session I didn't have a large enough sample size to prove that my visibly lopsided dice wasn't giving an equal probability distribution. So it seems obvious to me that no individual playtester would be in the position of being able to make a valid judgement about the average chance of failure in a challenge based on experience, rather than a calculation of probabilities, and that it'd take some pretty organized, extensive, and tightly-directed playtesting for that trend to become apparent in the aggregate playtest reports.

The point about level 1 and level 3 challenges giving different XP for overcoming the same DCs is obvious to the untrained observer, however, and that's still in there despite playtesting. <shrug>

 

[Wulf Ratbane]

It may be that most playtesters weren't in a position to actually check this out in depth.

I would never lay this at the feet of the playtesters.

Expecting a typical (target) 4e player to analyze the underlying math is like expecting a 1st level party to pass a complex skill challenge.

The fundamental promise of 4e is that the math just works and you don't have to worry about it. It's a good promise. It's right up there with "You don't need to know how to rebuild a carburettor to drive our car."

 

[Plane Sailing]

I would never lay this at the feet of the playtesters.

Expecting a typical (target) 4e player to analyze the underlying math is like expecting a 1st level party to pass a complex skill challenge.

The fundamental promise of 4e is that the math just works and you don't have to worry about it. It's a good promise. It's right up there with "You don't need to know how to rebuild a carburettor to drive our car."

Don't misunderstand me - I'm not laying this at the feet of the playtesters at all!

My concern is that WotC probably played their cards so tight to the chest that of the dozens of playtest groups, very few actually had even the opportunity to test this. I think that many of the playtest groups were probably testing scenarios rather than the rules.

Cheers

 

[Stalker0]

Ehy, Mr. Funny, I am a mathematician.
But since you are smarter than me, you can explain this:
1) We're interested in the number of failures before reaching a fixed number of successes.
2) The experiment consists of a sequence of independent trials.
3) The probability of success, p, is constant from one trial to another.

I'll be honest, until today I had no idea what a negative binomial distribution was. However, I looked it up on Wiki and this is what it told me. Please correct my ignorance if I am wrong.

It says that a NBD is a case where "each trial results in success or failure, the probability of success for each trial, p, is constant across the experiment and finally the experiment continues until a fixed number of successes have been achieved." (emphasis mine).

So the model doesn't care about number of failures, it just cares about when it hits the right number of successes.

Am I wrong?

 

[Plane Sailing]

BTW, I love mathematics like this.

I could watch it all day.

 

[Ander00]

The fundamental promise of 4e is that the math just works and you don't have to worry about it. It's a good promise. It's right up there with "You don't need to know how to rebuild a carburettor to drive our car."

Or with "multiclassing: any combo, any level, always works".

Anyway, kudos the the OP and everyone involved. Now if only WotC had done the math, or possibly, done the math again after changing some other rule that broke the one already in place.


cheers

 

[Wulf Ratbane]

My concern is that WotC probably played their cards so tight to the chest that of the dozens of playtest groups, very few actually had even the opportunity to test this.

WotC's design department is phenomenal, with talent and resources unmatched in the industry-- but it's still second place to "Open."

It's the same sort of great hubris at work that sees "professional journalists" decrying the work of bloggers.

 

[invokethehojo]

Stalker, let me say thank you.

I love the skill challenge system at heart and plan to use it a lot, but in my first few demos DM'ing my players hated it and said it was too hard. I listened, but I didn't hear till I read your math (thank god my college probability models class finally paid off somehow).

I am still gonna use skill challenges a lot, but I"m going with the 10/15/20 DC's, and some +2's here and there.

Also, for ability checks I think I'll go 7/10/13.

I think the creators made a good system, and it isn't hard to make tweaks at all, so I'm not gonna blame any of them. I"m just glad they gave me a system to work with at all