Why is the survey so broad then? You have a facile response for every problem we point out. That is not how you learn to do better.
(1) Survey is so broad b/c we're interested in the entire hobby games world rather than multiple separate-but-overlapping efforts that inevitably take more time and are more difficult to join for a holistic view of the overall hobby.
Any time you end up releasing a limited-focus hobby gams survey, what inevitably happens is that enthusiastic people will share it beyond the audience on which you're focusing, and you get a combination of extraneous responses and frustrated respondents. An example is the annual WSS survey that always gets shared beyond the miniatures wargaming crowd, which results in the board wargamers, fantasy skirmish minis gamers, and others, providing input of limited utility and complaining that it doesn't address their specific concerns.
We've tried to broaden ours to cover as many possible hobby gamers as we can, and there are plenty of gamers that overlap many different kinds of games. While I've got over 500 different RPG products on my shelves, I've also got several hundred wargames, over 100 Eurogames, multiple boxes of minis of all kinds. At this point in my life I'm primarily a wargamer, but I've been playing RPGs since '81.
(2) We're going to take all this feedback into future efforts and use it to improve later research. What we can't do is make mid-stream changes to something that's already got over 1300 responses, b/c that'll potentially invalidate what's already collected. We learned a lot from 2006 and used that feedback for this one. We'll use the feedback from this one to improve the next one (and I hope it won't be a 14-year gap again!). The issues raised above by the person with vision challenges is definitely useful and something to make sure we consider next time.
(3) As to responses for every issue you guys point out - I love the feedback b/c it helps us improve and shows that people care enough to get involved. So that's great! But if you guys are asking questions that we've already encountered or discussed or dealt with in previous survey research, should I
not respond? I'm making a good faith effort to engage so I can help explain our thinking behind what was done. Where we've screwed up, I think we've owned that (I'm not a fan of the Qualtrics UI, but I also don't have the option to have FDU change their survey software contract).
But there are other times where people bring up issues that were intentional (like the inverting of scales). Based on the comments, we might well go into a future study and decide "precedent be damned, we're changing this based on previous feedback". For now, though, there's a reason behind what we did. Would you prefer I not explain it?
