Very often open-ended questions in online surveys produce unsatisfactory results. This can be for various reasons. For example, respondents may misunderstand them or interpret them in a different way to that intended by the questionnaire designer. Or the questions may be phrased in such a way that they encourage limited answers.
But another important reason is speed. Many online panel survey respondents complete questionnaires at a speed which would surprise and disappoint the client. In doing so, they treat open-ended questions in the same way that they treat closed questions.
Open-ended questions tend to need more thought than many closed questions. If a respondent is asked their gender or marital status they do not need stop and think. But they usually do with open endeds. So the challenge is how to encourage them to do so.
Controlling the pace at which panel respondents can complete questionnaires is helpful in this regard. For example, we recently conducted a survey which included two open-ended questions in the questionnaire. We ran the soft launch without pace controls on the questionnaire, and then introduced pace controls for the main fieldwork.
The main fieldwork respondents wrote over twice the number of words in answer to the open-ended questions than did the soft launch respondents. There were over 100 soft launch respondents, so the sample size was reasonable for a comparison between them and the main fieldwork respondents. Moreover, the main fieldwork respondents’ answers were of pleasing quality.
How we control pace
Essentially, we hide the “Continue” button on certain pages of a questionnaire for a number of seconds. We also advise respondents at the beginning of questionnaires that we will be doing this.
The cost of controlling pace
One interesting element of this is that such controls do not typically add to the cost of surveys, or to the fieldwork period. This is partly because the addition of such controls reduces the amount of manual data cleaning that needs to be undertaken.
You might also like this article about how to get fuller responses to open-ended questions.
Or to find out more about running an online survey with MRQual please contact us.