How to stop online quantitative research grid straight-liners

There is a well-known problem with grid, or matrix, questions in online surveys. That is, respondents who “straight line” through them.

One way to try to resolve the problem is to include a grid element, or row, which cannot possibly have the same answer as another grid element. For example, if one statement in a grid is “Too much alcohol is bad for your health” and another is “Excessive alcohol consumption does not do much harm” then a respondent should not give the same answer (such as “Strongly agree”) to both. If they do, then they should ideally be shown a warning or excluded from the survey.

A slight variation on the above approach is to have two conflicting statements in different grid questions, rather than in the same one.

One problem with the above is that some respondents are wise to it. They know that if they give the same answer to each grid element then they could be excluded from the survey. Consequently, they may give random answers to different grid elements, rather than the same answer. Also, it does not resolve the fundamental problem, which is respondents speeding through grid questions. They only “straight-line” as a way of completing the grid question more quickly.

So, we use a different approach, which is a time-control. In other words, on a grid question we hide the “Continue” button for a certain number of seconds, depending on the number of words in the question.

When a respondent arrives at the grid question they see a message telling them about the time-control. In fact, at the start of the questionnaire we may also include a message about time-controls on various questions.

We have found that time-controls significantly improve the quality of the answers to grid questions. Respondents know they cannot speed through to the next question, so they may as well read the grid elements properly.

Leave a comment