The majority of responded statistically represents the majority of players.
We should look at selection bias. Because
@Micah Sweet is correct. The survey is not an accurate sample of the greater player base. We can also look at inferential statistics. In inferential statistics, if you want to know about the greater population, you can't just use any subset of people within that population.
A clear example from history is the 1936 US election for president. In that election you had two candidates, Alfred Landon and FDR. Literary digest conducted a large mail survey in this election. Literary digest was a very well-respected publication at the time. Their survey said that Landon would win with 57% of the vote. In reality, FDR got 61% and Landon got 37%. Over estimating Landon's support by 20%.
The error that Literary Digest encountered was selection bias. Specifically sampling bias. Sampling bias happens when your methods sample certain members of the population more than others. In the above example, Literary Digest used telephone directories, club memberships, and magazine subscriber lists. Research into the time period will immediately expose the issue at hand. Their sample was heavily weighted towards middle and upper class voters. The sources used for their mailing list were all based on luxuries that the lower class, largely, couldn't afford.
This is almost an identical situation. In a survey on D&DBeyond, WotC misses anyone who doesn't use D&DBeyond. A case study of sampling bias. If you were to poll only Enworlders, you might get the raw numbers needed for statistical significance, but your survey would not be representative of the player base as a whole. In both of the above cases, you lack a truly random sample, which means your data is unreliable when extrapolated to the greater population.
I hope that makes it clear why, in this case, the majority of respondents is not equal to the majority of players. The sample is riddled with errors. We could also get into non-response bias, if we needed more reasons, but I'll save everyone the pain.
TLDR: For the sample size to matter, your sample can't be riddled with errors in it's methodology.
EDIT: Just to be clear. Literary Digest had a sample of 2.2 million. Very much statistically significant.
Sources:
en.wikipedia.org
en.wikipedia.org
Two potential sources of error occur in statistical estimation—two reasons a statistic might misrepresent a parameter. Random error occurs as a result of
www.cliffsnotes.com
In recent Pivotal Research white papers , we’ve highlighted some common market research mistakes and offered remedies to avoid them. On paper, the benefits of
www.pivotalresearch.ca
