How to ensure the right people take your online surveys

People who have signed up to an online panel are regularly sent email invitations to take part in surveys. That’s why they sign up. They receive a reward, or incentive, for each survey they complete.

But very often a survey client will only want certain people to complete their questionnaire. This could be those who, for example, do most of their grocery shopping at a particular supermarket, who regularly undertake physical exercise, who smoke but are trying to give up, who own a dog, or whatever.

The way these people are identified in a survey is through “screener” questions at or near the beginning of a questionnaire. For example, if a survey is meant to be with people who undertake regular physical exercise then there might be a question early in the questionnaire which asks “Do you take regular physical exercise?” Anyone who answers “Yes” would be directed through to the next question so they can complete the questionnaire, and anyone who answers “No” would be screened out. That is to say, they would be thanked but excluded from the survey, and would not receive any reward.

The big problem with this is that panel members can get to know how the system works, and to recognise screener questions. And some of them will deliberately answer such questions in a way that (they hope) avoids being screened out.

But there is a solution. That is to ask screener questions in an indirect way, so that respondents cannot work out the qualifying criteria for completing surveys. For the example above of regular exercise this might mean that instead of a Yes/No screener question, there is instead a multi-response question about what activities respondents do at weekends or during the evening. This could include such options as shopping, going to the cinema, etc…

It can then be useful, if someone does answer that they do regular exercise, to have a follow-up question which asks what exercise they do, where they do it, etc… Ideally this would be open-ended.

Even better, ask people an open-ended question before the (closed) screener question(s) about what activities they do on a typical weekend or evening. Then ask the closed question(s) but include an instruction that they should not give any answer that they did not write down in the previous open-ended.

The level of care taken to ensure that survey respondents are genuine is a matter for each client. But undoubtedly, the more care that is taken, the more useful will be the survey results.