Lanefan
Victoria Rules
I've felt "snowed" on some of their surveys and been suspicious of the rest.I think I did... there were a number of their surveys/playtest surveys I responded to during that time (due to the 2024 playtest packets) and so I likely did that one too.
Yeah, it's hard sometimes to know what the design parameters of the surveys are, or how to interpret the questions, when we're getting different numbers of questions based on partially randomization and partially based on answers provided early in the survey. Doubly so if/when the surveys were designed by some third-party firm who has more experience with marketing research for soft drinks or action figures and playsets than for TTRPGs. Plus that perhaps that they felt (correctly or not) they already had enough of a sense of the desires of more aged players. FWIW, in all of their surveys I took I never felt deliberately or intentionally snowed.
What's more telling to me is that whenever they do a survey and there's a thread about it here, how vastly different the surveys we each saw and did seem to be. The cynic in me says it's because they already 99% know what results they want and if some early answers from a respondent indicate those "correct" answers aren't forthcoming that respondent doesn't get the rest of the relevant questions.
A truly honest survey would:
a - ask exactly the same questions of every respondent, with every question having a complete range of possible answers including variably positive, variably negative, and neutral
b - every question has a "not applicable" option (e.g. 4e-related questions for someone who never played 4e would be n/a, but you still get to see those questions anyway)
c - every question (or at least every group of questions) has an open-ended comments section
d - no "trap" questions, i.e. those where the same thing is asked in a different way in order to get a different response
e - compile and include every response they get; the only discarded responses being duplicates from the same person or from bots
f - release the actual survey results to the public - as in, the hard numbers plus comments - along with the summary.