4E playtesting or lack thereof

[Fifth Element]

I'm happy to accept that they had minimal, cursory and utimately inadequate play testing.

And I am certainly not intending to attribute incompetence, or worse, to any game designers. There are lots of pressures in publishing a new game, and mistakes are inevitable.

Doesn't "mistakes are inevitable" render this whole discussion moot? I agree that mistakes are inevitable in something with this degree of complexity. So what's the issue?

I think I agree with Obryn that playtesting may not have caught this perceived problem.

 

[wayne62682]

Are we really seriously discussing something asserted on a message board by someone calling himself Trollman?

Not only that but Frank Trollman is infamous on RPG messageboards for basically outright saying "WotC's designers are a bunch of idiots who couldn't design a game to save their lives". He's pretty much been banned from nearly every RPG board out there (along with a group of his cronies who frequent a board called The Gaming Den where they curse out WotC and laugh at people who don't min-max) because he gets outright hostile and insulting when somebody doesn't agree with him.

 

[Obryn]

I think I agree with Obryn that playtesting may not have caught this perceived problem.

I'd say it's more than a perceived problem!

However, it's one of those problems which will only appear when you have a rather large sample of skill challenges for each playtest group. When you consider all the other vagaries that go into it - the DM giving situational bonuses, Aid Another actions, allowances made for good roleplaying, powers which increase skill bonuses - I think it's murky at best.

I know there's a big tendency to believe that playtesting solves all problems with a game, but I completely disagree. I think playtesting is important, but it's not the end-all, be-all of overall product testing. Playtesting is awesome for feedback like, "This mechanic isn't fun" or "We didn't understand these rules; they need more explanation" or "This part seems awkward. Can you make it better?"

I think it's a very poor tool to find what are, in the end, mathematical and statistical issues. Gaming groups have been coping with and glossing over these since the dawn of gaming, and I don't know if playtest groups necessarily turn this part of their brains off.

So, I guess what I'm saying is that I completely disagree with the assertion that thorough playtesting "obviously" would have found the mathematical problems. I think that mathematical analysis would have, though, and I'm not sure what happened with the final table. My guess is eleventh-hour changes, but I'm not really in a position to know.

-O

 

[Rechan]

You also have to consider the possibility that the playtesters agree with the philosophy of the designers with regards to the rules. A philosophy that the average gamer doesn't buy into.

For instance, take the Sure Strike/Careful Attack issue. Many people on the boards have complained about the veritable weakness of those two AWs. When Mearls was asked about it on a podcast, he and Noonan said "Yeah well, we consider Accuracy far more important than damage."

Even if they have playtesters playtesting the Sure Strike/Careful attack, if the Playtesters also hold the same opinion as the designers (accuracy is more important than damage), then the problem isn't going to get ironed out because they don't perceive it as a problem.

 

[Doug McCrae]

WotC... has not released a version that is actually used by the designers in their home games

Exactly like 1e AD&D, then.

 

[amethal]

Not only that but Frank Trollman is infamous on RPG messageboards for basically outright saying "WotC's designers are a bunch of idiots who couldn't design a game to save their lives". He's pretty much been banned from nearly every RPG board out there (along with a group of his cronies who frequent a board called The Gaming Den where they curse out WotC and laugh at people who don't min-max) because he gets outright hostile and insulting when somebody doesn't agree with him.

If you don't want to discuss this topic, then don't.

Otherwise, feel free to play the ball rather than the man.

 

[Fifth Element]

I'd say it's more than a perceived problem!

By perceived problem, I don't mean something that some people think is a problem, but really isn't. I mean it's something that some people perceive to be a problem, and others don't.

I don't perceive it to be a problem, but I can't say others are wrong for seeing it as a problem. That's why I qualified the word.

The system is playable as written. Some find the math to be screwy, so it's a problem for them. Not trying to discount their view, just saying it's not objectively a problem. Not that anything in RPG mechanics could really be objective.

 

[Fanaelialae]

Mathematically, skill challenges as presented in the DMG are rough, but not unworkable. That is, you can run skill challenges with the old DCs without your game exploding, and the rules are quite simple to follow. I'm a bit surprised the math wasn't scrutinized more closely, but when it comes down to it, you could still run successful encounters with the final system. If a lot of the concentration was on higher-complexity skill challenges, it's not surprising the playtesters found more success than failure.

Regardless, I don't know that this particular problem is something that playtesting would have caught. Having run a few pre-revision Skill Challenges, they actually do work (as in, they are playable) it's just the success rate is low at certain levels. I think it's better-suited to mathematical analysis, like Stalker0 ran, and that playtesting isn't the right tool to find its flaws.

I also don't know what sorts of revisions the table went through. Were the DC as-published? Was the footnote about skills there originally? I'm honestly not sure.

-O

This.

You have to consider that a 1st level character's skill modifier can range anywhere from +15 (+5 training, +3 focus, +5 "20 ability score", +2 racial bonus) to -1 (-1 "8 ability score"). Finding the "sweet spot" between those ranges was no doubt tricky, more so because the designers essentially had to guess just how challenging the majority of gaming groups would prefer them.

Mathematical models like Stalker0's are interesting, but they are based on assumptions that don't necessarily apply to all groups (they're a "best guess" as to what an average group might look like). Hence, the reliability of mathematical predictions deviates in relation to how much one's own group differs from the mathematical "default" group.

I wouldn't be surprised if the designers set the DCs based on the assumption that DMs would more likely than not custom-tailor skill challenges to their group (meaning that the group would have high modifiers more often than not). Under that assumption, the DCs work fine. For non-tailored challenges, the errata DCs work better.

I don't think that skill challenges were the most rigorously playtested element of the 4e rules. IIRC, one of the designers mentioned that the skill challenge concept was a relative latecomer to the design of 4e.

Nonetheless, I seriously doubt that they weren't playtested. I've yet to see real evidence to back that claim.

 

[Fifth Element]

Otherwise, feel free to play the ball rather than the man.

Generally words to live by, but in this case the ball is insubstantial, making it difficult to play with. There's the promise of a ball, but we haven't really seen one yet.

 

[ggroy]

I think it's a very poor tool to find what are, in the end, mathematical and statistical issues. Gaming groups have been coping with and glossing over these since the dawn of gaming, and I don't know if playtest groups necessarily turn this part of their brains off.

So, I guess what I'm saying is that I completely disagree with the assertion that thorough playtesting "obviously" would have found the mathematical problems. I think that mathematical analysis would have, though, and I'm not sure what happened with the final table. My guess is eleventh-hour changes, but I'm not really in a position to know.

If WotC is using a secret mathematical formula to check new crunch for balance, I assume they could be running computer Monte Carlo simulations of some sort on the new crunch. If done correctly, the Monte Carlo simulations will run through thousands of cases and tell you whether a set of new rules is on average overpowered or underpowered, and how wide the variance is.