4E playtesting or lack thereof

[Fifth Element]

Originally Posted by amethal

I'm happy to accept that they had minimal, cursory and utimately inadequate play testing.

And I am certainly not intending to attribute incompetence, or worse, to any game designers. There are lots of pressures in publishing a new game, and mistakes are inevitable.

Doesn't "mistakes are inevitable" render this whole discussion moot? I agree that mistakes are inevitable in something with this degree of complexity. So what's the issue?

I think I agree with Obryn that playtesting may not have caught this perceived problem.

[wayne62682]

Originally Posted by Fifth Element

Are we really seriously discussing something asserted on a message board by someone calling himself Trollman?

Not only that but Frank Trollman is infamous on RPG messageboards for basically outright saying "WotC's designers are a bunch of idiots who couldn't design a game to save their lives". He's pretty much been banned from nearly every RPG board out there (along with a group of his cronies who frequent a board called The Gaming Den where they curse out WotC and laugh at people who don't min-max) because he gets outright hostile and insulting when somebody doesn't agree with him.

[Obryn]

Originally Posted by Fifth Element

I think I agree with Obryn that playtesting may not have caught this perceived problem.

I'd say it's more than a perceived problem!

However, it's one of those problems which will only appear when you have a rather large sample of skill challenges for each playtest group. When you consider all the other vagaries that go into it - the DM giving situational bonuses, Aid Another actions, allowances made for good roleplaying, powers which increase skill bonuses - I think it's murky at best.

I know there's a big tendency to believe that playtesting solves all problems with a game, but I completely disagree. I think playtesting is important, but it's not the end-all, be-all of overall product testing. Playtesting is awesome for feedback like, "This mechanic isn't fun" or "We didn't understand these rules; they need more explanation" or "This part seems awkward. Can you make it better?"

I think it's a very poor tool to find what are, in the end, mathematical and statistical issues. Gaming groups have been coping with and glossing over these since the dawn of gaming, and I don't know if playtest groups necessarily turn this part of their brains off.

So, I guess what I'm saying is that I completely disagree with the assertion that thorough playtesting "obviously" would have found the mathematical problems. I think that mathematical analysis would have, though, and I'm not sure what happened with the final table. My guess is eleventh-hour changes, but I'm not really in a position to know.

-O

[Rechan]

You also have to consider the possibility that the playtesters agree with the philosophy of the designers with regards to the rules. A philosophy that the average gamer doesn't buy into.

For instance, take the Sure Strike/Careful Attack issue. Many people on the boards have complained about the veritable weakness of those two AWs. When Mearls was asked about it on a podcast, he and Noonan said "Yeah well, we consider Accuracy far more important than damage."

Even if they have playtesters playtesting the Sure Strike/Careful attack, if the Playtesters also hold the same opinion as the designers (accuracy is more important than damage), then the problem isn't going to get ironed out because they don't perceive it as a problem.

[Doug McCrae]

Originally Posted by Windjammer

WotC... has not released a version that is actually used by the designers in their home games

Exactly like 1e AD&D, then.

[amethal]

Originally Posted by wayne62682

Not only that but Frank Trollman is infamous on RPG messageboards for basically outright saying "WotC's designers are a bunch of idiots who couldn't design a game to save their lives". He's pretty much been banned from nearly every RPG board out there (along with a group of his cronies who frequent a board called The Gaming Den where they curse out WotC and laugh at people who don't min-max) because he gets outright hostile and insulting when somebody doesn't agree with him.

If you don't want to discuss this topic, then don't.

Otherwise, feel free to play the ball rather than the man.

[Fifth Element]

Originally Posted by Obryn

I'd say it's more than a perceived problem!

By perceived problem, I don't mean something that some people think is a problem, but really isn't. I mean it's something that some people perceive to be a problem, and others don't.

I don't perceive it to be a problem, but I can't say others are wrong for seeing it as a problem. That's why I qualified the word.

The system is playable as written. Some find the math to be screwy, so it's a problem for them. Not trying to discount their view, just saying it's not objectively a problem. Not that anything in RPG mechanics could really be objective.

[Fanaelialae]

Originally Posted by Obryn

Mathematically, skill challenges as presented in the DMG are rough, but not unworkable. That is, you can run skill challenges with the old DCs without your game exploding, and the rules are quite simple to follow. I'm a bit surprised the math wasn't scrutinized more closely, but when it comes down to it, you could still run successful encounters with the final system. If a lot of the concentration was on higher-complexity skill challenges, it's not surprising the playtesters found more success than failure.

Regardless, I don't know that this particular problem is something that playtesting would have caught. Having run a few pre-revision Skill Challenges, they actually do work (as in, they are playable) it's just the success rate is low at certain levels. I think it's better-suited to mathematical analysis, like Stalker0 ran, and that playtesting isn't the right tool to find its flaws.

I also don't know what sorts of revisions the table went through. Were the DC as-published? Was the footnote about skills there originally? I'm honestly not sure.

-O

This.

You have to consider that a 1st level character's skill modifier can range anywhere from +15 (+5 training, +3 focus, +5 "20 ability score", +2 racial bonus) to -1 (-1 "8 ability score"). Finding the "sweet spot" between those ranges was no doubt tricky, more so because the designers essentially had to guess just how challenging the majority of gaming groups would prefer them.

Mathematical models like Stalker0's are interesting, but they are based on assumptions that don't necessarily apply to all groups (they're a "best guess" as to what an average group might look like). Hence, the reliability of mathematical predictions deviates in relation to how much one's own group differs from the mathematical "default" group.

I wouldn't be surprised if the designers set the DCs based on the assumption that DMs would more likely than not custom-tailor skill challenges to their group (meaning that the group would have high modifiers more often than not). Under that assumption, the DCs work fine. For non-tailored challenges, the errata DCs work better.

I don't think that skill challenges were the most rigorously playtested element of the 4e rules. IIRC, one of the designers mentioned that the skill challenge concept was a relative latecomer to the design of 4e.

Nonetheless, I seriously doubt that they weren't playtested. I've yet to see real evidence to back that claim.

[Fifth Element]

Originally Posted by amethal

Otherwise, feel free to play the ball rather than the man.

Generally words to live by, but in this case the ball is insubstantial, making it difficult to play with. There's the promise of a ball, but we haven't really seen one yet.

[ggroy]

Originally Posted by Obryn

I think it's a very poor tool to find what are, in the end, mathematical and statistical issues. Gaming groups have been coping with and glossing over these since the dawn of gaming, and I don't know if playtest groups necessarily turn this part of their brains off.

So, I guess what I'm saying is that I completely disagree with the assertion that thorough playtesting "obviously" would have found the mathematical problems. I think that mathematical analysis would have, though, and I'm not sure what happened with the final table. My guess is eleventh-hour changes, but I'm not really in a position to know.

If WotC is using a secret mathematical formula to check new crunch for balance, I assume they could be running computer Monte Carlo simulations of some sort on the new crunch. If done correctly, the Monte Carlo simulations will run through thousands of cases and tell you whether a set of new rules is on average overpowered or underpowered, and how wide the variance is.

[Jack99]

Originally Posted by ggroy

In a post by Windjammer in the closed thread "WotC Strategy of Planned Obsolescence?", an issue of playtesting 4E stuff was brought up.

http://www.enworld.org/forum/4842667-post17.html

I vaguely recall something about the RPGA being used to playtest 4E before it was released. I remember reading older Dragon magazine articles which mentioned older editions also used the RPGA for playtesting, such as for 2E AD&D.

Does anyone know how much playtesting was actually done on older splatbooks, campaign settings, etc ... back in the 1E/2E AD&D and 3E D&D days?

Some of the links in Windjammer's article refers to articles by FrankTrollman suggesting that very little to no playtesting was actually done on later 3.5E splatbooks, such as "The Book of Nine Swords".

If it turns out that the 4E splatbooks had very little to no playtesting done on them before publication, it would be interesting to see how robust 4E is to overpowering/underpowering and whether balance is determined by a publicly unknown (ie. proprietary WotC) secret mathematical formula run on all new crunch rules.

Hi.

Do you play 4e?

[Fanaelialae]

Originally Posted by ggroy

If WotC is using a secret mathematical formula to check new crunch for balance, I assume they could be running computer Monte Carlo simulations of some sort on the new crunch. If done correctly, the Monte Carlo simulations will run through thousands of cases and tell you whether a set of new rules is on average overpowered or underpowered, and how wide the variance is.

I'm quite certain there is no secret, magical formula (beyond perhaps, +1/2 level for players and +level for monsters, which is hardly a secret).

This is WOTC, not the Pentagon.

[ggroy]

Originally Posted by Jack99

Hi.

Do you play 4e?

Yes.

[Cadfan]

Wizards of the Coast: Almost a year ago, we released a fully playable playtest version of the Artificer that was about half the size of the intended final version. Thousands of you read it, played it, and emailed us feedback or discussed it with our designers on our forums. Based on your comments and your experiences, we've made the following changes, and are pleased to announce the release of the full, final version of the Artificer.

This Thread: The final version isn't the same as the playtest version. There's no evidence at all that the released version of the Artificer was playtested in any way.

[Windjammer]

Originally Posted by Obryn

OK - then what did you mean by the "priceless" thing?

See above. I have absolutely no interest to engage in attributing or declaring tacit intentions, be they other people's or my own.

Originally Posted by Fifth element

What sort of playtesting do you consider "solid"?

I guess that type of question is easier to answer in the negative, and even then easier to answer by example rather than principle. Supposing I were to pitch a new class or racial write up to 4E Dragon, building a PC with that class/race once and seeing him through a couple of combats would probably not do to ensure that my class/race is compatible with the remainder of the system. What matters, in the end, is not some absolute quantity of time and rigor that goes into playtesting but whether that time and rigor is enough to ensure that the product being playtested doesn't cause widespread problems in the customer base upon releasing it.

Does that help you understand my claim, or do you still think it's too hazy? (If so, let me know which part in particular you find elusive.)

Originally Posted by Fifth element

The wording - "no single version" - also discounts the possibility of solid playtesting of the system as it evolved during playtesting. What do you consider a "single version"?

I said "single version released by WotC in June 2008 to June 2009". So "release" is the key word. Official WotC releases include: a physical product released by WotC, like a rule book, an offering on D&D Insider, or an errata download on the WotC webpage. (If I'm inadvertently overlooking possibilities of official releases, let me know!)

[malkav666]

Hello,

I read this thread and the source thread written by the OP. It would seem to me that 4e and its progression of rules and the level of balance is about on par with the previous edition.

Yes there are elements in both 4e and 3e that I think were hastily released, and perhaps did not receive as much oversight and testing as they deserved. But due to the modular nature of both systems, it is fairly easy to strike unbalanced deliveries from the game.

As for the specific item of skill challenges in 4e I would say that no they were not perfect. back when my group was giving 4e a whirl the thing that really disappointed us about them had nothing to do with any specific mechanical aspect of the system itself. It was that it seemed that so much of the non-combat 3e stuff was grouped into this single mechanic. But aside from that complaint I did not find the system to be unplayable. we actually had a fairly exciting skill challenge once about the group trying to drive a sled down a mountain in pursuit of another sled.

Sure, I think it could have used some more work, but it is my opinion that the assertion that the mechanic was not playtested all is false.

love,

malkav

[Windjammer]

Originally Posted by Cadfan

Wizards of the Coast: Almost a year ago, we released a fully playable playtest version of the Artificer that was about half the size of the intended final version. Thousands of you read it, played it, and emailed us feedback or discussed it with our designers on our forums. Based on your comments and your experiences, we've made the following changes, and are pleased to announce the release of the full, final version of the Artificer.

This Thread: The final version isn't the same as the playtest version. There's no evidence at all that the released version of the Artificer was playtested in any way.

Wizards of the Coast: Almost a year ago with the relase of the 4E DMG, we released a playtest version of the skill challenges. Thousands of you read it, played it, and emailed us feedback or discussed it with our designers on our forums. Based on your comments and your experiences, we've made the following changes, and are pleased to announce the release of the full, final version of the skill challenge subsystem. Here's your DMG 2 (that will be $29.90 please) - OR - here's your free download at our website.*

This Thread: uuhhm, okay. I just wish you had written that on the blurb of the DMG 1 before I bought it.

-----

Cadfan, in all seriousness, there's a world of a difference between material like the Artificer or the Barbarian, classes which become part of 4E as soon as their playtest versions are released on DDI, and stuff like hybrid-classes which are banned from RPGA play for a reason - their being half-baked and having not nearly seen enough playtesting for WotC to say "yep, we're fine with it, some tweaking left to do, but nothing that will cause trouble".

* I mean that "OR". I'm just as curious as anyone else here which format WotC will choose to re-release the skill challenge subsystem (should they choose to do so).

[Obryn]

Originally Posted by ggroy

If WotC is using a secret mathematical formula to check new crunch for balance, I assume they could be running computer Monte Carlo simulations of some sort on the new crunch. If done correctly, the Monte Carlo simulations will run through thousands of cases and tell you whether a set of new rules is on average overpowered or underpowered, and how wide the variance is.

I wouldn't be surprised at all if they do run simulations, but I'd guess it's far from top-secret. The math is clearly all there, just waiting to be crunched. I know that many of the designers have really good heads for math, as demonstrated on some of the podcasts.

Regardless, though, like I said - I have no idea what particular ball got dropped for skill challenges. It could have been eleventh-hour changes, it could have been ivory tower design... I have no idea. I don't think you can attribute it to playtesting, though. It's not a hammer that can pound all nails.

-O

[tomBitonti]

Originally Posted by amethal

Thus either WotC didn’t playtest the skill challenges which appeared in the DMG, or they chose to publish something they knew was wrong, or the playtesters failed to spot something that a great many other people (apparently) spotted straight away.

Originally Posted by Fifth Element

Now, this is different than an assertion that #1 is the case. You've offered three options, which I think cover the possibilities pretty well.

To me, #3 seems the most likely. I personally don't have any issues with the system in the DMG, but I know many people do. Maybe the designers just saw it the way I do, who knows.

We don't really have any evidence to support any one of these three possibilities, so if you are going to assert that a particular one is correct, you need to provide the evidence when you do so.

That "which appeared in the DMG" is too strong of a qualifier. Switching from "Skill challenges" to "Skill challenges which appeared in the DMG" is a big difference.

I can easily see that skill challenges were play tested, but, in the mad rush to put out the DMG, the test results were not fully integrated into the rules. Or, they found a lot of problems, and had to make a quick decision to get the product out.

That to me looks like true problems in the play test (in which I am including steps to integrate the results into the final rules). I think that is more arguable than to posit that there was no play testing at all.

[Obryn]

Originally Posted by Windjammer

See above. I have absolutely no interest to engage in attributing or declaring tacit intentions, be they other people's or my own.

Eh? You were saying that people shouldn't attribute motives to you, so I wasn't, and honestly don't care in regards to the thread as a whole. But what's wrong with asking for clarification on something you wrote? It's just a weird comment. I mean, was it intended as an insult?

What matters, in the end, is not some absolute quantity of time and rigor that goes into playtesting but whether that time and rigor is enough to ensure that the product being playtested doesn't cause widespread problems in the customer base upon releasing it.

Was it your impression that the released skill challenge mechanics caused widespread problems?

I agree they were mathematically wacky, and that this wackiness should have been caught. They don't look like they're broken at first glance, but a thorough breakdown shows that they don't do quite what you'd expect them to do. But do you believe peoples' home games were negatively affected by the rules as released to an extent that a gaming group not currently engaged in statistical analysis would even notice?

My contention is that the rules are, indeed, broken, but not in such a way that a gaming group - including a playtest group - would necessarily notice.

-O