D&D General Is D&D Survey Feedback Read? [UPDATED!]

If you watch a lot of YouTube videos, you may be aware that there's a narrative going around, with 'anonymous' sources that contain Machievellian quotes about how WotC ignores survey feedback, and uses it as some kind of trap to keep discussion off the internet.

We're all unhappy with WotC and its approach to the current licensing situation, and we're all concerned about the fate of the third-party D&D publishing industry which supports hundreds, if not thousands, of creators and small publishers. I'm worried, and afraid for the fate of my little company and those who rely on me to pay their rent, bills, and mortgages.

But it's important to stay factual.

Ray Winninger, who ran D&D until late 2022, said "I left after the first OneD&D feedback was arriving. I know for certain UA feedback is still read."

He went on to say "This is simply false. Before I left WotC, I personally read UA feedback. So did several others. Many, many changes were made based on UA feedback, both quantitative and written. The entire OneD&D design schedule was built around how and when we could collect feedback."

Winninger previously spoke out in support of the OGL movement, after WotC announced their plans in December.

Another WotC employee tweeted, too -- "I read nearly half a million UA comments my first year working on D&D. I was not the only one reading them. I understand the desire to share information as you get it, but this just feels like muckraking."

It's important to stay on the right side of this OGL issue -- and make no mistake, any attempt to de-authorise the OGL is ethically and legally wrong -- but just making stuff up doesn't help anybody.

Benn Riggs, author of Slaying the Dragon: A Secret History of Dungeons & Dragons, chimed in with his own suspicions.

Here is why I am growing more and more suspicious of @DnD_Shorts and their purported source in WoTC. Let's call that source "The Rogue."

1) Getting a source on the record takes time. DnD Shorts is getting commentary incredibly quickly. WoTC's statement came out this morning, and by this afternoon, we know "The Rogue's" thoughts. The statement talks about a survey? "The Rogue" tells us no one will read what we write to the company.

Then there is the logistics. Is "The Rogue" contacting DnD Shorts from WoTC HQ? Doing it from the bathroom? On their lunch break? All while knowing they'd be fired if found out? They don't at least wait to contact DnD Shorts from home?

2) The info provided by "The Rogue" is simply too good. They have mentioned where they work in the company, and directly quoted powerful people within the company. All that means that within WoTC, tracking down "The Rogue" and firing them should take about two hours. Frankly, if "The Rogue" exists, the best proof of it will be when they are fired.

I'm upset about the OGL too, and it's easy to cast doubt on anonymous sources. People have done it to me. So I will say upfront I could be totally wrong about this and if DnD Shorts reads this and curses me for a bastard because they're honest & good & true and I am besmirching them, well I'm sorry.

But something here just feels wrong, and I cannot keep my peace.

And of course, all this fracturing of the 'resistance' only weakens the position of those who are working against the de-authorization of the OGL. The more click-bait nonsense out there, the less seriously anybody takes the real issues which affect real people.

UPDATES! WotC designer Makenzie De Armas has weighed in to describe the survey process:

Hi, actual #WotCStaff and D&D Designer here. I am credited on several UA releases—and I’ve made edits to that content based on both qualitative and quantitative survey results. Let’s walk through what happens behind the scenes of a UA, shall we?

1. We design player-facing mechanical elements that we hope to include in a future product. We then place those mechanical elements into a UA document and release it, to see what our player base at large thinks of it.

2. We release a survey about the UA.

3. The survey information is collated by members of the team. It’s broken down into two parts: quantitative satisfaction expressed as a percentage, and a summary of qualitative feedback trends noticed in the comments.

4. That summary is reported back to the product teams. The designers on the product teams then make edits to the mechanical elements based on the feedback summary.

5. If satisfaction doesn’t meet our quality standards, we’ll rerelease mechanical content in a followup UA.

This is a proven process. Take for example the Mages of Strixhaven UA, where we tried to create subclasses that could be taken by multiple classes. (Fun fact: that was my first UA.) Did we, as studio designers, want that to work? Yes! But it didn’t.

And we learned that it didn’t BECAUSE of the UA process. We learned that it wasn’t something a majority of our players wanted; we also learned what small elements of that design DID bring joy. We salvaged those elements, redesigned them, and put that changed design in the book.

If we didn’t read or listen to feedback, we would have put those polyclass subclasses into the final book, and the product would have been worse for it. Yes, of course we want to know if you like something—we’re game designers! We’re creating something that is meant to be FUN!

And yes, sometimes we get frustrated when people tell us how to do our jobs, or use those feedback opportunities to belittle us; we’re human. But despite all that, we’re still going to listen and always strive to improve. That’s the truth.

They went on to say:

When I say ALL the comments, I mean it in the most literal sense. We have team members who have dedicated WEEKS to diligently reading through feedback. It’s honestly incredible, and I applaud my team members’ work!

Gamehome Con director Alex Kammer added:

Hey everyone. I personally know the guy at Wizards whose job in part is to read and organize all the comments from their surveys. Reasonable OGL talk and demanding action is great. Fallacious hit pieces only cause harm.

 

[ad_hoc]

Assuming a 5 day work week, they would have read almost 2,000 comments per day, every single work day. That's 240 comments read per hour for 8 hours a day.

So they're claiming that they read 1 comment every 15 seconds for 8 hours straight, for a whole year.

Yeah, I'm not sure I believe that.

Most of the comments are going to be similar.

I used to do quality analysis for a background screening company.

I would be given huge stacks of forms for police screening checks. My job was to look for errors from data entry. Mostly typos, sometimes unclear writing, and rarely invalid requests.

I went through them far quicker than you would probably guess. Once I was in the zone I could scan entire documents in seconds.

I bet I could read 100 comments a minute for the short ones.

 

[EzekielRaiden]

"I won't do the actual effective way of expressing my feelings as a way of sticking it to the man" is counterproductive.

The problem is, WotC is the one with a trust deficit. They have to prove to us that this channel is, in fact, "the actual effective way of expressing [our] feelings." It may in fact be the case that this accusation is bunk and they genuinely do read every single comment and truly try to understand what each person is saying. I personally find that suggestion ludicrous, given the amount of time one would need to invest in doing so--based on reports from the current staff denying the allegation, this is ONE PERSON'S job. One single person had to filter through, what, 38000+ comments for the "One D&D" surveys? And you expect me to believe this person had time to do literally anything else?

The ball is in WotC's court. It has been for a long time. They need to make an actually concrete gesture that will earn back some trust. Unless and until they do, they will remain in a trust deficit. We need something real, not platitudes.

Besides, several of the claimed debunking statements from current WotC personnel don't actually refute the core claim. They specifically focus on the numerical metrics (which...yes, that's literally what the allegation claimed, that the feedback is only collected to get "temperature checks" on things) and that some employees filter out the comment stuff, which the actual designers never look at. Not one thing I've seen thus far contradicts any of that. Moreover, not one thing I've seen refutes the claim that the data gets manipulated, buried, or emphasized in order for the designers to justify doing whatever it is they were already intending to do....which is an accusation I've been making since at the very least 2013 during the Next playtest....as I said earlier.

 

[Scribe]

This is not the point.
If everyone tells them even going back all the way does not help, they can as well burn all the bridges to you and 3pp and just cater to the 95%* of the people who don't use 3pp anyway.

*those people only playing and not hanging around in geek forums all the time.

You discount the value in not creating people who are radically opposed to the direction of a company.

Those people talk, for a long long long time, about how they have been wronged.

 

[dave2008]

Telling us "we will not revoke the OGL and we will make that promise legally-binding with all due speed" is trivial. Do you seriously mean to say that a statement to the public is some horribly difficult thing that takes three weeks to happen?

But they don't want to do that. They want to revoke it. So they have to be more nuanced. Now, we finally we will finally get a draft and an official channel to respond soon. So that is where you can tell them how they have messed up - and I plan too!

 

[Whizbang Dustyboots]

The problem is, WotC is the one with a trust deficit. They have to prove to us that this channel is, in fact, "the actual effective way of expressing [our] feelings."

OK, no.

If your wife cheats on you, and you thus have an extremely reasonable trust deficit, you shouldn't then start being suspicious of her when she mentions the law of gravity or the sun rising in the east.

For any purpose with any company, dogpiling a Facebook or Twitter thread is a terrible way to communicate. I know a bunch of social media managers, and even the most dedicated ones just grab a handful of representative quotes and pass them on.

Using a feedback form means your responses get fed into a spreadsheet or a database and then keywords can be pulled easily to gauge sentiment. In WotC's case, we have a bunch of employees who tell us those forms are then printed out in some fashion and actually read.

That's simply not going to happen with Twitter or Facebook because, even with tools to make it easier to work with, pulling massive amounts of comments that way for consumption off-line is hard, even for people with a lot more advanced training than the folks at WotC's comms, who tend to just be super-fans and not library science majors or former data journalists.

 

[LordEntrails]

But making a press release wherein you say, "We hear you, we understand that we've done wrong, and we are committed to respecting your demands. We will not revoke the OGL 1.0a, and we will prepare a legal statement to that effect as soon as we can."? That's not complex. That's not difficult. There are a million ways to do that, and it takes all of a couple days for the person-in-charge for this subject to make a determination and prepare a statement.

You're assuming that the corporation can make a decision that quickly. It's unlikely that their is widespread agreement throughout WotC as to what the decision should be. Hence it takes time for different factions to voice their opinions, for a consensus to be built, for people to fret about the decision, and to then have whoever is high enough up to say "Fine, this is the decision we are going with."

This is a big screw up, It is pretty obvious that different parts of WotC and Hasbro have different opinions about a path forward. It does not surprise me it has taken so long for someone to step up and make a public statement. Every screw up from here on out is going to impact somebody's career (as it should). You don't get too many people to step up and make such decisions when there is not widespread consensus.

 

[Scribe]

If your wife cheats on you, and you thus have an extremely reasonable trust deficit, you shouldn't then start being suspicious of her when she mentions the law of gravity or the sun rising in the east.

No, but I'm going to question every time she doesnt mention who's going to be there when she goes out with 'friends'.

Just as Wizards fails to mention the OGL when they mention "we hear you and are adjusting X".

 

[Mistwell]

Because WotC (and, more importantly, Hasbro) is a corporation, and corporations have an extremely serious problem with becoming collusions against the customer?

You have the actual people, by name, directly giving you the information about this, all re-posted in this thread. They're not corporate suits, they're not high up executives, they are front line people who directly do this job telling you how the job is done, why, and how genuinely offensive it is to be telling them (not some nebulous big corp you have a hate-on for, but them personally as people doing their actual job) that they're not doing the job they do day in and day out.

When you get that level of detailed information, AND THEN THE LEAKER HIMSELF WALKS BACK HIS ACCUSATIONS, you pause. You pause and re-assess the information you have on hand. You do not just bully ahead and stick to your guns when the facts change, you take a moment and think about it.

And let's be utterly clear:

1) The leaker posted this today, "But, it seems that these insiders were incorrect on this unless more information comes to light. "

2) The "inside source" also walked back their comments about none of it being read when challenged.

 

[Bayushi_seikuro]

Does anyone else find it to be an incredibly bad idea for them to show the actual reply they got?

I think it may have been the wrong move to show the actual reply they got back. In addition to the person's writing style, we know they've seen the reports based off of One D&D feedback, but they're only "aware" of (as opposed having seen) how previous surveys were handled. This is information that can help Wizards figure out who's been talking.

I know my company has found leakers with less to go on.

Exactly this. And also quoting specific conversations, while transparent, can definitely tip where the leak is. Mark Hamill told the story on Graham Norton that Irvin Kershner came to him with the Vader reveal and told him, "George knows this. I know this. And when I tell you, you'll know this. And if it gets out, we'll know you were the one." i think that was the interview when he also used the phrase "Telegraph, telephone, tele-Carrie."

 

[EzekielRaiden]

OK, no.

If your wife cheats on you, and you thus have an extremely reasonable trust deficit, you shouldn't then start being suspicious of her when she mentions the law of gravity or the sun rising in the east.

For any purpose with any company, dogpiling a Facebook or Twitter thread is a terrible way to communicate. I know a bunch of social media managers, and even the most dedicated ones just grab a handful of representative quotes and pass them on.

Using a feedback form means your responses get fed into a spreadsheet or a database and then keywords can be pulled easily to gauge sentiment. In WotC's case, we have a bunch of employees who tell us those forms are then printed out in some fashion and actually read.

That's simply not going to happen with Twitter or Facebook because, even with tools to make it easier to work with, pulling massive amounts of comments that way for consumption off-line is hard, even for people with a lot more advanced training than the folks at WotC's comms, who tend to just be super-fans and not library science majors or former data journalists.

You are assuming that I expect these things to act as an effective communication channel. I don't, because they won't. That isn't their purpose.

Their purpose is to make sure that WotC knows their reputation is still getting shredded. That we haven't forgotten how mad we are, that we aren't just going to move on because the news cycle has focused on something else.

THAT is what it is for. It is, very literally, to make sure WotC can't just pretend that the controversy has blown over and they don't need to do anything. It's saying, "Your delay tactics won't work."

Because, again, the demand is simple, and if they don't know what it is, they're already ignoring feedback to begin with.

That doesn't mean I won't use it. I will! I will totally use it, and I will tell WotC exactly, in no uncertain terms, how upset I am about their behavior and what they need to do to fix it. If I didn't consider it wrong, I'd even entertain the idea of making multiple accounts so I could fill out the survey multiple times.

But I will not--absolutely, positively will not--let them turn this into a way to silence and dismiss rightful anger by hiding it behind the wall of corporate silence. If the survey is badly-written, I will express that. If their new OGL is still flawed, you'd better believe I'll keep doing my teeny tiny part to continue making waves.

I bow not yet before the Iron Crown,
nor cast my own small golden sceptre down.