This is a true story about a closed multi-response question in an online panel survey that we ran.
Respondents were asked to select the personal care products they used regularly, from a list of 19 different options.
The question text and answers together comprised of exactly 100 words. At an average reading speed of 200 words a minute, we would therefore expect respondents to take roughly 30 seconds to complete this question. This, of course, assumes that they do not need to think about their answers.
Speed of response
We found that out of the first 100 respondents, 49 answered in 4 seconds or less. And 87 answered in less than 10 seconds. Just 4 respondents took 30+ seconds. This shows that barely any of the respondents can have read the question or answered it properly.
With this particular survey we manually cleaned out these records, before then putting in place the solution below.
The standard industry approach to the problem of speeders is to exclude the fastest 5%-10% of respondents. But based on this example, such an approach is clearly inadequate, to say the least.
A better solution is to include a time-control on such questions. This means that respondents will not be able to move to the next question for a certain number of seconds.
We have found that this does not lead to a higher drop out of respondents, or to increased fieldwork time. In addition, it need not lead to higher costs, because there is less manual data cleaning needed at the end of the survey.
This was just one example, but we have seen many similar ones. Basically, if you let respondents race through questionnaires then many (or most) of them will do so.