4 essential tips for designing and executing better online research

This post is sponsored by Milieu Insight.

Online market research, or online surveys, are now one of the most used methodologies in the broader market research industry. As marketers are under increased pressure to run more campaigns with shorter lead times, there’s greater demand for fast insights that can keep up with the pace of digital marketing today.

But marketers need to ensure that speed doesn’t come at the expense of the quality or depth of insight. Minor errors in survey design can result in major issues with data quality, which in turn, can lead your business to take the wrong course of action.

So to help you get the most out of your investment in research here are four essential tips to help you design and execute better online surveys.

Keep your surveys short

One of the cardinal sins of online surveys today is running a questionnaire that’s too long. This often stems from a desire to leave no stone unturned by throwing in every possible question. Other times this can happen when a researcher doesn’t adjust their mindset when switching between different methodologies such as qualitative versus quantitative.

For example, when you run a focus group or in-person interview, a 45 minute to hour-long engagement is pretty common, and completely acceptable. But the interaction an interviewee will have in a qualitative environment doesn’t translate to a screen. You simply can’t expect a respondent to take a 30-minute survey without getting fatigued. 

Our very own Dr Antarika Sen ran a controlled experiment on Milieu’s panel and found that 10 minutes (or roughly 50 questions) is the breaking point. Anything longer and you start to see the tangible effects of respondent fatigue (that is, more non-committal responses in open-ends, increased likelihood of selecting neutral options on ordinal scales, etc). If you’d like to learn more about this you can read about Antarika’s study approach and results here. 

The main takeaway is that you need to keep your surveys short and concise. We generally advise our clients to aim for 20 questions or less. We also pre-collect more than 1,000 demographic, attitudinal and behavioural questions that we can use to enrich their results, turning a 10-question study into more than 1,000 questions. This helps our clients feel more at ease when running shorter survey engagements.

Don’t impose a high cognitive load

Aside from long surveys, another issue we often come across is surveys and questions that impose a high cognitive load. This can include questions with very long response lists (for example, more than 15 response options).

But a major culprit of high cognitive load is a grid/matrix question. If you’re not familiar with grids, they are essentially a series or single or multi-select questions stacked on top of each other (example below). They may be efficient for the researcher who is programming the survey, but they almost always result in a poor user experience for the survey taker. Even Qualtrics advises against them. 

Image Source: ​​https://www.questionpro.com/features/matrix-spreadsheet-survey-question.html

Whether we’re talking about grids, long response lists, or just really complex question types, you need to consider the usability of your survey. The more cognitive load you impose, the higher the chances your respondents may fatigue and start phoning in responses.

Don’t lead, bias or steer respondents

Online survey-based research is a business grounded on asking “good” questions. Simply put, words matter, and failing to communicate a question clearly can lead, bias or steer your respondents, resulting in poor data quality. There are some fundamental principles you can follow when it comes to crafting well-written survey questions, but this is where it helps to have a great research partner.

We have a team of researchers that help our clients with activities such as survey design for custom research projects. But we’ve also developed a library of ready-to-use survey templates, available via our customer dashboard that covers the most common use-cases. 

Caption: Milieu Canvas Dashboard

If you’re not using a survey template and plan to design a questionnaire from scratch, it always helps to consult someone who has experience in online research to sense-check your survey before it goes live.

Avoid unnecessary screening

If you’ve ever taken a paid online survey, chances are you’ve seen a question like this:

This is actually a staple-screening question in the research industry. The assumption here is that anyone who works in market research, or in an industry related to the subject matter of the survey, will bias the results. Well, I’ve got news for you, these can do more harm than good.

Survey takers join panels to get rewards, and any panellist who has taken more than one survey on a panel knows that if they select anything aside from “None of the above” they’ll get screened out. This causes panellists to start gaming surveys, resulting in poor data quality.

We have a “no panellist left behind” policy, where every respondent who starts a survey gets points, which helps cultivate a value exchange where respondents don’t try to game our surveys. If a client is concerned about respondents who work in a certain industry, we have the ability to pre-target respondents using more than a thousand pre-collected profiling data points.

But no matter who your research partner is, you need to consider whether screening questions such as this are necessary, as they may actually be hurting your data quality.