D&D (2024) The WotC Playtest Surveys Have A Flaw

[Umbran]

And that fraction of a fraction of a fraction of people that made it all the way to Step (e) is supposed to be everyone's voice in the room.

Let us be clear - since they claim to specifically read all the comments made, if literally everyone showed up, they couldn't handle the responses in reasonable time. So, no, it isn't supposed to be everyone.

I don't have a better idea, but still. That's a big ask.

This is why the survey results also have to go through developers. It isn't development by democracy.

 

[Njall]

Look, while I'm not really a 5e fan, I appreciate them playtesting the game, because getting a free preview of the game is nice, if nothing else.

That said, the main purpose of a playtest isn't (or shouldn't be) reaching a consensus among the playerbase. Sure, when something isn't well received, the devs tend to either correct it or explain the reasons behind the changes, but the main reason behind a playtest is finding problems and ironing out the bugs.
Also, considering the way the playtest packets have been released, with each packet presenting only a few elements rather than the complete picture of how the new rules are supposed to interact with each other, I'd argue than the process is flawed for more reasons than just the number of people actually taking the time to answer the surveys.

I'm not sure any of this matters to WotC, tho, since both 5e and One D&D's playtests were probably about garnering goodwill from the community, rather than anything else. It doesn't really matter if the amount of feedback they get is adequate, what matters is that they're actively asking for feedback and reacting accordingly.
Getting useful feedback in the process is probably just a nice bonus.

 

[Parmandur]

That said, the main purpose of a playtest isn't (or shouldn't be) reaching a consensus among the playerbase. Sure, when something isn't well received, the devs tend to either correct it or explain the reasons behind the changes, but the main reason behind a playtest is finding problems and ironing out the bugs.

Worth noting that WotC has a large internal playtest network that does that sort of rigorous testing: the UA process is for "taste testing" to see if significant portions of the broader playerbase like something or not before they iron out the details and bugs.

Also, considering the way the playtest packets have been released, with each packet presenting only a few elements rather than the complete picture of how the new rules are supposed to interact with each other, I'd argue than the process is flawed for more reasons than just the number of people actually taking the time to answer the surveys.

The packets already point to the bigger picture: the 2014 PHB. All they have tested are new plug and play options for the same fram, as noted in the packets.

 

[grimmgoose]

Juat as an aside, the evidence suggests that DMs do almost all the book buying: there aren't any 5E products that aren't essentially DM products.

I've run two groups (multiple systems, settings, campaigns, etc.) for a combined total of a decade at this point. They have purchased zero (0) rulebooks for any campaign we've run. They have no problem pitching in to help me purchase a book ("hey, I want to run Lancer. Would anyone be interested? We could split the rulebook?") But they aren't going to shell out money for a book that they will reference once-per-week (if that - more and likely, they'll use it once to build a new character and that's it).

WOTC's current strategy of, "every book is for everyone" has lost me. When I look at how much I've spent on 5E this past year vs Savage Worlds, the difference is staggering. I've only kept my D&D Beyond sub, but I've thrown hundreds of dollars at the SW kickstarters. They just provide more value for me. WOTC's books don't, full stop.

It's making me stop pulling groups to 5E. I want to play systems that make me feel supported.

 

[Njall]

Worth noting that WotC has a large internal playtest network that does that sort of rigorous testing: the UA process is for "taste testing" to see if significant portions of the broader playerbase like something or not before they iron out the details and bugs.

Well, if internal testing were enough to iron out every bug, the game wouldn't have had errata and a fair amount of clarifications over the past few years.
Even the 0.01% of their fanbase is, quite likely, an order of magnitude or two bigger than their internal playtest team and allows for better results in a shorter amount of time.

The packets already point to the bigger picture: the 2014 PHB. All they have tested are new plug and play options for the same fram, as noted in the packets.

The amount of stuff they're changing ( i.e. the addition of weapon mastery, the revamped feats, stuff like the rogue getting "cunning strike" in addition to sneak attack) isn't really insignificant, in terms of balance.
Asking us to only playtest chunks of the rules in isolation doesn't seem like the better course of action, if they're still in the process of revamping the basic framework of the game, because that "bigger picture" you're pointing at is also subject to change.

For example, the revamped feats, which we were presented in one of the first packets, included a modified version of "Great Weapon Master".
All the feedback provided about how the new feat , which was given when we could only evaluate the new version in the context of the 2014 PHB, becomes less and less valid as new versions of the base classes, especially the martials, are released.

Until they provide a cohesive, comprehensive playtest of how the new version of the game is supposed to be played, any feedback we might provide is, at best, inaccurate.

 

[Parmandur]

WOTC's current strategy of, "every book is for everyone" has lost me.

I think

Well, if internal testing were enough to iron out every bug, the game wouldn't have had errata and a fair amount of clarifications over the past few years.
Even the 0.01% of their fanbase is, quite likely, an order of magnitude or two bigger than their internal playtest team and allows for better results in a shorter amount of time.



The amount of stuff they're changing ( i.e. the addition of weapon mastery, the revamped feats, stuff like the rogue getting "cunning strike" in addition to sneak attack) isn't really insignificant, in terms of balance.
Asking us to only playtest chunks of the rules in isolation doesn't seem like the better course of action, if they're still in the process of revamping the basic framework of the game, because that "bigger picture" you're pointing at is also subject to change.

For example, the revamped feats, which we were presented in one of the first packets, included a modified version of "Great Weapon Master".
All the feedback provided about how the new feat , which was given when we could only evaluate the new version in the context of the 2014 PHB, becomes less and less valid as new versions of the base classes, especially the martials, are released.

Until they provide a cohesive, comprehensive playtest of how the new version of the game is supposed to be played, any feedback we might provide is, at best, inaccurate.

But they aren't revamping anything other than what theybare testing: the 2014 PHB is fundamentally the same game, only the details are changing. And for the early experimental tests, they didn't know what changes would stick, so how could they present them as a whole...?

 

[TheSword]

I’m ambivalent about the survey. I check out the playtests but I don’t do the surveys. I don’t think you can create art by committee.

To be honest some elements of 5.2 I will like, some I probably won’t. But every edition has been the same. So far it seems better

If it’s much worse and too great a departure we’ll switch to something else like we did around 4e. Winners find a way.

In the grand scheme of things, a couple of changes to an RPG have very impact on the game, let alone on life. I’m not surprised your gamer friends don’t really care.

 

[Njall]

But they aren't revamping anything other than what they bare testing: the 2014 PHB is fundamentally the same game, only the details are changing. And for the early experimental tests, they didn't know what changes would stick, so how could they present them as a whole...?

They're changing classes. That's not "details". Weapon Mastery, for example, is a whole new system.
The way feats interact with classes is very significant, balance wise.

"Well, the new version of Great Weapon Fighter seems more in line with the rest of the feats, but hits fighters, who tend to rely on it for a good chunk of DPS, pretty hard" becomes moot if, say, you change the fighter class so that its DPS isn't reliant on the feat anymore.

They should either provide the full extent of the changes, or at least try and group together the subsystems that tend to have a decent amount of interaction.

It's not like they didn't know that they intended to work on the rest of the game as well when they released the first packets.

 

[Parmandur]

They're changing classes. That's not "details". Weapon Mastery, for example, is a whole new system.
The way feats interact with classes is very significant, balance wise.

"Well, the new version of Great Weapon Fighter seems more in line with the rest of the feats, but hits fighters, who tend to rely on it for a good chunk of DPS, pretty hard" becomes moot if, say, you change the fighter class so that its DPS isn't reliant on the feat anymore.

They should either provide the full extent of the changes, or at least try and group together the subsystems that tend to have a decent amount of interaction.

It's not like they didn't know that they intended to work on the rest of the game as well when they released the first packets.

Classes are plug-in options for the system, yes. You could re0lace them all without changing the rules per se (the actual rules change proposals are in the Rules Gallery).

But they aren't looking for that level of detail in feedback, it's a satisfaction survey: how much do you like this option on a 1-5 scale? DPS or charop analysis isn't what UA is collecting. It's a te.perature test to see if people like an idea, before they do the hard crunch playtest on the math.

 

[bert1001 fka bert1000]

Well, if internal testing were enough to iron out every bug, the game wouldn't have had errata and a fair amount of clarifications over the past few years.
Even the 0.01% of their fanbase is, quite likely, an order of magnitude or two bigger than their internal playtest team and allows for better results in a shorter amount of time.

Yeah I really don't understand why you wouldn't want to use free playtesting these days.

I'm always amazed by the crazy fan made in depth analysis that comes out 2-3 weeks after new material. Math layed out, great comparison charts, pro and cons, etc. Even elegant solutions to problem areas.

I guess it might hurt sales if you give out the 90% final version as a final playtest? But I don't think so. There are easy ways to understand the final changes without buying the book.

These days, you buy the book because you want to play with the changes. 30 years ago you'd have to buy the book to know what the changes were.