D&D 4E Ben Riggs' "What the Heck Happened with 4th Edition?" seminar at Gen Con 2023

[Thomas Shey]

Based on the numbers involved, they have a more than representative sample, yes.

Numbers, per se, don't actually tell you that, if the selection method already biases things. This comes up regularly in the way some surveys are using out of date methodologies (phone surveys still often use land line numbers, which is going to end up aiming at A) People who still have land lines, and B) Pick them up rather than let an answering machine get them. That's going to fish in only a limited pond, but it can still produce relatively large numbers if the poll taking is extensive).

 

[Parmandur]

It's not the principle I'm unsure of, it's the execution.

Results seem to speak for themselves.

 

[Parmandur]

Numbers, per se, don't actually tell you that, if the selection method already biases things. This comes up regularly in the way some surveys are using out of date methodologies (phone surveys still often use land line numbers, which is going to end up aiming at A) People who still have land lines, and B) Pick them up rather than let an answering machine get them. That's going to fish in only a limited pond, but it can still produce relatively large numbers if the poll taking is extensive).

They are limiting themselves to people willing to sign up for Beyond...but that's like 13 million people and growing, so not exactly a small set. About 600,000 have participated in the surveys, which is more than adequate as a sample size of the customer base.

 

[Thomas Shey]

They are limiting themselves to people willing to sign up for Beyond...but that's like 13 million people and growing, so not exactly a small set. About 600,000 have participated in the surveys, which is more than adequate as a sample size of the customer base.

Again, only if you don't assume the people signing up for Beyond are pre-selecting in a way that does a poor representation of the market as a whole. Not to mention those who will respond to a passively presented survey.

 

[Parmandur]

Again, only if you don't assume the people signing up for Beyond are pre-selecting in a way that does a poor representation of the market as a whole. Not to mention those who will respond to a passively presented survey.

13 million people is basically the market as a whole, yes. That's also why they have focused on onvosrding people to Beyond in their social media, and making it as painless as possible. Filtering out people not on Beyond is part of filtering the noise, as you suggested would be a good idea.

 

[Micah Sweet]

Again, only if you don't assume the people signing up for Beyond are pre-selecting in a way that does a poor representation of the market as a whole. Not to mention those who will respond to a passively presented survey.

To be fair, since by far the most important metric to WotC for the surveys success is sales (which is doingbpretty well), what would it matter to them if the survey is poorly representative of the community? They're getting what they want (more money) right now.

 

[Alzrius]

Results seem to speak for themselves.

The results will be what the finished product looks like, and how well it's received not just upon release, but after a few years of use by the public. Until we get to that point, it's too soon to reach any sort of conclusion.

 

[Micah Sweet]

13 million people is basically the market as a whole, yes. That's also why they have focused on onvosrding people to Beyond in their social media, and making it as painless as possible. Filtering out people not on Beyond is part of filtering the noise, as you suggested would be a good idea.

This implies that they don't want and/or care about D&D fans who aren't on Beyond. Is that what you're saying?

 

[Parmandur]

To be fair, since by far the most important metric to WotC for the surveys success is sales (which is doingbpretty well), what would it matter to them if the survey is poorly representative of the community? They're getting what they want (more money) right now.

Well, they have a sale motivation to get a representative sample of the market and follow the money. There isn't a difference between the two: pursuing money successfully is identifying the community of purchasers, researching their desires, and servicing them.

 

[Parmandur]

This implies that they don't want and/or care about D&D fans who aren't on Beyond. Is that what you're saying?

Why would WotC possibly care about people who refuse to sign up for their free portal...? They are bending over backwards to make it easy to sign up, no strings attached, and putting out tons of free stuff on there. At this point, refusal to sign up for Beyond is absolutely a safe indicator of not being part of the representative sample of people who will spend money on D&D.

I don't use Beyond for anything, but I signed up years ago for the free stuff. It takes like ten seconds.