4E playtesting or lack thereof

[Cadfan]

Originally Posted by Windjammer

Cadfan, in all seriousness, there's a world of a difference between material like the Artificer or the Barbarian, classes which become part of 4E as soon as their playtest versions are released on DDI, and stuff like hybrid-classes which are banned from RPGA play for a reason - their being half-baked and having not nearly seen enough playtesting for WotC to say "yep, we're fine with it, some tweaking left to do, but nothing that will cause trouble".

But there's no difference at all for the purposes of my post, which is about the logical shiftiness going on here:

Regardless of context, it is ridiculous to claim that no playtesting occurred based on changes made after playtesting. Defining the only playtesting you will accept as being playtesting of "the final version" is just a silly little dodge. It enables the speaker to almost always claim that no "real" playtesting occurred because playtesting almost always leads to changes and tweaks, therefore rendering the playtested version "not final."

And for a sufficiently bitter person, if playtesting occurs which leads to no changes at all, therefore satisfying their demand that the final version be the playtested version, they can claim that the playtesting was pointless because it was obviously ignored.

[Mallus]

Originally Posted by ggroy

Yes.

Does 4e seem badly playtested to you?

Bonus questions: did you play prior editions of D&D? Did they seem badly playtested to you?

[malraux]

I also suspect that the multiple versions of skill challenges result not from refinements of the mathematical elements, but from changes in what skill challenges are perceived to do. Consider: Should SC be on level with most encounters (in that most parties will likely win) or should SC be something with a higher risk/reward? That changes what you want the math to do quite a bit.

Should SC be resolved quickly, or be part of an ongoing multisession event? I'm sure there are multiple other dimensions SC could be analyzed on as well. The point is that these aren't necessarily questions that can be answered via play testing. Its an overall design question. More importantly, there aren't guaranteed to be right answers.

To me SCs look like they underwent several design goal changes, rather than poor playtesting or mathematical analysis. In addition, after being released to the public, I think the feedback was that many players saw the goal of SCs as different from what the designers predicted.

[catsclaw227]

WindJammer -- I think it's tough to look at the ball and not see the man too. The Frank Trollman post wasn't the best example for your cause and I still don't know what the goal of his post was... Was it to call out WOTC designers? Was it to express his concern with the playtesting of skill challenges? Was it to stick a wrench into the gaming enjoyment of the 4e players that are just fine with the skill challenges and/or the errata presented?

Pointing to his post as your evidence (or maybe it was for collaboration on a thought you already had?) only emphasized the potential for this to be a flame/editionwar/complain thread.

I agree that the skill challenges as presented in the DMG needed some work. We ran them a few times prior to the errata and, though they seemed tough, the game fun didn't diminish by their inclusion. Actually, the game was better for it.

Skill Challenges are the right idea. The implementation is proving to be more esoteric than I think anyone could have guessed. Between all the Ruling Skill Challenges articles, forum posts that describe examples in play *of which there are MANY good examples here on EnWorld*, the alternate systems proposed (Stalker0), and the individuality of the DMs that acutally use them, there appears to have been many observations about what works for any particular game table and what doesn't.

How I apply challenge complexity, difficulty (low, medium, high), circumstance bonuses and other modifiers, and how I percieve a success or fail all play into this system. There are a lot of assumptions that can change whether the base skill challenge system is balanced or not. Is this a problem with the core system, maybe, but its much more an issue around how the game is played at the table.

Originally Posted by Obryn

Was it your impression that the released skill challenge mechanics caused widespread problems?

I agree they were mathematically wacky, and that this wackiness should have been caught. They don't look like they're broken at first glance, but a thorough breakdown shows that they don't do quite what you'd expect them to do. But do you believe peoples' home games were negatively affected by the rules as released to an extent that a gaming group not currently engaged in statistical analysis would even notice?

My contention is that the rules are, indeed, broken, but not in such a way that a gaming group - including a playtest group - would necessarily notice.

At my table, we've had no problems with them. The players love the way they work, they all feel involved, failure isn't a terrible thing (as it's usually just a stumbling block), and that it gives a much better experience than "roll once at a DC, problem solved." At my table, they are embedded in roleplaying challenges and have been short or long in game sequences. We recently ran a skill challenge that took up 3 hours of game time, as they gathered clues around a city. It was went really well for all, and I got props after the game.

As Obryn stated above, the playtesting itself could have been a factor if the playtesters agreed with the assumptions the designers had.

Do you think that OD&D or 1E had the level of playtesting that went into 4e? I doubt it, seriously. Yet no one is writing threads like this about their favorite old school game.

This isn't life or death and a math fix for the skill challenge system won't hurt anyone. Nor has it impacted the games I play or run. Has the skill challenge system math caused you to give up on 4e?

Is the point of this thread to state that a complex game has a subsystem that didn't quite work as well with 1 million players as it did with 500 playtesters? Don't all games suffer from the same problems?

Or was it to state without any proof that WOTC never tested skill challenges, neener, neener, neener, how-do-you-like-them-apples?

[ggroy]

Originally Posted by Mallus

Does 4e seem badly playtested to you?

Bonus questions: did you play prior editions of D&D? Did they seem badly playtested to you?

4E combat doesn't appear to be excessively overpowered or underpowered, at least for most of the stuff in PHB1. I haven't got around yet to playing any classes or races from PHB2. I haven't used any of the crunch from the Martial Power book yet. The stuff in Arcane Power for wizards doesn't seem to be overpowered or underpowered at this point. Though I haven't tried out every single new thing for the wizard yet. For the stuff I've tried so far, it appears to be balanced.

For prior editions, 1E AD&D seemed to have clunky rules for psionics in the PHB. They didn't seem to be well thought out. No idea how well they playtested it.

I do remember 1E AD&D magic users being very powerful at high levels. Though whether that was a deliberate design goal or a flaw, I don't really know.

For 3.5E, I never got the opportunity to create any munchkin overpowered characters. The DMs I played with banned almost all splatbooks and mainly allowed the core books only. Most of the 3E/3.5E campaigns I played in, the DM ended the game at around level 8 or 9. For the stuff we played (ie. lower than level 10), there didn't seem to be any major overpowering issues I can recall. I can't comment on higher levels, since I never got the opportunity to play any. I felt that the 3E core rules were probably reasonably well playtested for lower levels (ie. less than 10), though this is probably not saying much.

[AllisterH]

I think one of the reasons why skill challenges playtesting might not be caught is that this is one of the cases where even if the PCs fail the challenge, the actual challenge itself is a success for the players involved and (most importantly), even though the skill challenge itself wasn't successful, the outcome isn't a game ender.


Reading the comments about skill challenges (both good and bad) over the past several months, one of the common traits is that a good or bad skill challenge seems to have absolutely nothing to do with whether or not the players actually completed the SC successfully.

[Windjammer]

Originally Posted by catsclaw227

WindJammer -- I think it's tough to look at the ball and not see the man too. The Frank Trollman post wasn't the best example for your cause and I still don't know what the goal of his post was... Was it to call out WOTC designers? Was it to express his concern with the playtesting of skill challenges? Was it to stick a wrench into the gaming enjoyment of the 4e players that are just fine with the skill challenges and/or the errata presented?

For one last time: there's no hidden agenda, let alone a "cause" behind anything. As far as I'm concerned (I can't speak for others, obviously), Frank's post(s) argue that the amount of playtesting for skill challenges was unacceptable, from a customer's point of view, and I concur. Whence the need to pin an agenda to the making of this claim? It's straight forward customer feedback, the like of which you can read on any customer feedback forum, be it for electronic devices, car engines, or what have you.

As to the question why quoting that particular instance of customer feedback is salient in this thread, I can really just point you to the thread title. To be honest, it strikes me as seriously bizarre when I have to justify why I see fit to quote the material I did, and other people address all sorts of points (including my motives in doing so, over and over and over again) but not the one named in the thread title.

Originally Posted by catsclaw227

Is the point of this thread to state that a complex game has a subsystem that didn't quite work as well with 1 million players as it did with 500 playtesters? Don't all games suffer from the same problems?

Or was it to state without any proof that WOTC never tested skill challenges, neener, neener, neener, how-do-you-like-them-apples?

Are you seriously asking? After I've stated three times over what my intended, demonstrative claim is? And then offering this travesty of a question?

Originally Posted by Cadfan

Regardless of context, it is ridiculous to claim that no playtesting occurred based on changes made after playtesting.

Where, in this thread, did I (or, for that matter, anyone else) claim this?

Originally Posted by Cadfan

Defining the only playtesting you will accept as being playtesting of "the final version" is just a silly little dodge. It enables the speaker to almost always claim that no "real" playtesting occurred because playtesting almost always leads to changes and tweaks, therefore rendering the playtested version "not final."

Why do you think that? I'm curious. Personally, I would have thought that designers playtest material until they hit a version during the playtest - not after - when they say "yep, we're happy with that". In fact that's exactly my impression of a great amount of the combat system (and individualc lass powers) in 4E. The only point of contention from my side, actually, is to point out this discrepancy.

Just to mention a salient example. Know how some Daily powers in 4E have the keyword “reliable”? Well, thank the playtesters for that because WotC designers didn’t put it there before playtesters alerted to them that it’s hugely anti-climactic to have one’s daily’s (or at least some thereof) misfire. Guess what, WotC inserted the keyword, other playtesters confirmed that this was a good change, and so the final version kept it.* That, in a nutshell, is how playtest ought to work. Anything else ISN’T a playtest, it’s post-playtest adjustments to material.


*By the way, this is based on evidence of me having talked to a 4E playtester. I haven't been a playtester myself.

[Mallus]

Originally Posted by ggroy

4E combat doesn't appear to be excessively overpowered or underpowered, at least for most of the stuff in PHB1. I haven't got around yet to playing any classes or races from PHB2.

My group does uses material from most of the books. So far we've had to edit out some things, but by and large we find 4e balanced.

I do remember 1E AD&D magic users being very powerful at high levels. Though whether that was a deliberate design goal or a flaw, I don't really know.

Both.

For 3.5E, I never got the opportunity to create any munchkin overpowered characters. The DMs I played with banned almost all splatbooks and mainly allowed the core books only.

All you need is the 3.5e core books.

Most of the 3E/3.5E campaigns I played in, the DM ended the game at around level 8 or 9.

3.5e works well at those levels. Well, except the parts that don't, like multiclassed spellcasters, but on the whole sub-10th level 3.5e is a very good system.

However, as may be germane to the original topic, it's easy to point to parts of 3.5e and claim it was insufficiently playtested (for instance, by pointing out that multiclassing spellcasters --especially in a core-only game-- doesn't work).

[Mallus]

Originally Posted by Windjammer

That, in a nutshell, is how playtest ought to work. Anything else ISN’T a playtest, it’s post-playtest adjustments to material.

What it is, is a bug fix. A patch.

This is the way product releases in the real world work. Early adopters are part of the beta test.

[ggroy]

Originally Posted by Mallus

3.5e works well at those levels. Well, except the parts that don't, like multiclassed spellcasters, but on the whole sub-10th level 3.5e is a very good system.

In one of my previous 3.5E games, the DM was receptive to continuing on after level 9, but with a caveat that he would ban any and all prestige classes. Unfortunately half of the players objected to this, and the game was brought to a final end. The DM thought that most prestige classes were either overpowered and/or outright silly.

[catsclaw227]

Originally Posted by Windjammer

For one last time: there's no hidden agenda, let alone a "cause" behind anything. As far as I'm concerned (I can't speak for others, obviously), Frank's post(s) argue that the amount of playtesting for skill challenges was unacceptable, from a customer's point of view, and I concur. Whence the need to pin an agenda to the making of this claim? It's straight forward customer feedback, the like of which you can read on any customer feedback forum, be it for electronic devices, car engines, or what have you.

As to the question why quoting that particular instance of customer feedback is salient in this thread, I can really just point you to the thread title. To be honest, it strikes me as seriously bizarre when I have to justify why I see fit to quote the material I did, and other people address all sorts of points (including my motives in doing so, over and over and over again) but not the one named in the thread title.

Well, apparently I am not the only one that thinks the Frank Trollman had an additional agenda, so it must not be odd of a question. He doesn't just argue the points. He wants to see mearls with crow in his teeth. He flat said this. So, it would be irresponsible to NOT question his motives because for 99% of the world, especially when it's a forum thread, a persons motives color their arguments.

Yes, the thread title asks if there was actual playtesting done on the 4e skill challenge system. But you do refer to his post, right at the beginning, as evidence that it wasn't. It is not bizarre in a debate to question the sources of a position. IANAL, but isn't that one of the things trial lawyers do?

I am also not the first person to question his statements as a basis of your argument. It must not be that bizarre to ask about it like I did.

For what it's worth, I like some of your takes on other 4e things, as read in your posts in other threads, but in my opinion, your claim isn't well supported for this one, so I question it.

Originally Posted by Windjammer

Are you seriously asking? After I've stated three times over what my intended, demonstrative claim is? And then offering this travesty of a question?

I know what your stated claim is. The subtext in the argument is what I am questioning. I apologize for being snarky with the "neener,neener" part, but my first question was more what I was hoping you would state.

Isn't all this about the fact that, as I stated, a complex game has a subsystem that didn't quite work as well with 1 million players as it did with 500 playtesters? And that most all RPGs suffer from this issue?

What about the rest of my points? Do you have any comments on that, including how, in actual play, the skill challenges haven't been a problem?

Is the goal of your post simply to let us know how you feel about skill challenge playtesting? Are you looking for confirmation from WOTC? Is it to attempt to fix the problem and find a solution instead of having all of us grumble about it?

I know what you stated thesis is... what is the goal of the post?

[ExploderWizard]

Originally Posted by Mallus

What it is, is a bug fix. A patch.

This is the way product releases in the real world work. Early adopters are part of the beta test.

While this is true, it is an unacceptable way to test physical product unless replacements of "updated" content are given free of charge. Software can use that model much easier because the product is much simpler to patch.

It's like those bank commercials on TV. Even kids know that holding out on someone isn't very nice.

[AllisterH]

*Shrug*

If Blizzard can't produce a perfect product, no company can. Blizzard is well known in the industry as having the most extensive playtests around.

From their some time months long internal playtesting to BETAs that last several weeks, you would figure Blizzard would catch everything.

Hell no.

First month after release of the Lich King, the new death knight class gets some tweaks.

Some times playtesting will reveal a fault only after much time has passed. Take the monk for example. When the monk was first released, for months it seemed afterwards, did people call it overpowered.

Same thing with the Big Six items. When 3.0 was first released, the big six were not even a glimmer in the eyes of most players but these emerged after a couple of years

[catsclaw227]

Originally Posted by Windjammer

That, in a nutshell, is how playtest ought to work. Anything else ISN’T a playtest, it’s post-playtest adjustments to material.

You are absolutely right about this. But like software, most game systems requires some patching to catch the things that a million users will see that 500 playtesters did not.

Errata is a real world adjustment. This really shouldn't be anathema to the players of RPGs, but we sure like to complain a lot about it.

Maybe because i am a software developer, I am softer on these kinds of issues...

[Obryn]

Well, I'll say that I think it's completely irrelevant why Windjammer posted the thread. I don't really care. I'm still looking for clarification on one or two comments, but whatever.

I am, however, still puzzled as to why playtesting is considered the end-all be-all of product testing; or why it's taken as a given that playtesting - even thorough playtesting - will catch all a game's flaws. I don't see a reason to believe that this is the case, and I can't recall an example of a complex game in which this was true.

Are there any?

-O

[ggroy]

Originally Posted by Obryn

I am, however, still puzzled as to why playtesting is considered the end-all be-all of product testing; or why it's taken as a given that playtesting - even thorough playtesting - will catch all a game's flaws. I don't see a reason to believe that this is the case, and I can't recall an example of a complex game in which this was true.

Are there any?

-O

Chess?

[Mallus]

Originally Posted by ExploderWizard

While this is true, it is an unacceptable way to test physical product unless replacements of "updated" content are given free of charge.

But errata are free of charge.

As to 'enhancements' to the system which really address issues in the original design... well, they're going to cost you money. Note this is currently more true for the previous edition than 4e. A lot of later additions to 3e where meant to address various issues with the rule set, and for my money, much of that later material was good.

Software can use that model much easier because the product is much simpler to patch.

Now that's funny! Software patches may be easier to disseminate, they're not easier to write.

[Fifth Element]

Originally Posted by Windjammer

Where, in this thread, did I (or, for that matter, anyone else) claim this?

From the OP, quoting you in another thread:

Originally Posted by OP

As he points out time and again, WotC has NOT playtested a single version of skill challenges to date (June 2008-June 2009) and has not released a version that is actually used by the designers in their home games, whether in a core book for which they charged $30 or an similarly pricey online service. Which is disheartening.

(emphasis added)

Technically, you did not claim this in this thread. It was in another thread, which was the inspiration for this thread.

[Mallus]

Originally Posted by ggroy

Chess?

Chess was an open-source project with a development cycle that lasted roughly 1500 years.

[ExploderWizard]

Originally Posted by AllisterH

*Shrug*

If Blizzard can't produce a perfect product, no company can. Blizzard is well known in the industry as having the most extensive playtests around.

From their some time months long internal playtesting to BETAs that last several weeks, you would figure Blizzard would catch everything.

Hell no.

First month after release of the Lich King, the new death knight class gets some tweaks.

Software is a different animal. What if blizzard charged extra for the patches or using the patches required you run little sub programs along with the application ( stuffing paper errata sheets into a book)? Unless patches are seamless to the end user they are a burden and contribute to product failure.