Double-barreled and Dichotomous: 6 Common Mistakes in Survey Question Writing

By the Editors

Writing effective survey questions is an essential part of getting the business insights that you need to make important strategic decisions. But this process isn’t just about understanding your research agenda and optimizing your question format choices for different devices. Applying survey writing best practices can help you avoid common mistakes that can damage the integrity of your data. Let’s take a look at some of the most common mistakes in survey writing and tips for avoiding them.

Follow the Funnel in Question Order

 

Question order plays an important role in overall survey design. Understanding these dynamics helps when you’re thinking through your survey layout and question development. On one level, it’s helpful to avoid an overall organization that might bias respondents later in the survey. At the same time, researchers rely on question order to provide respondents to online surveys or mobile market research initiatives with enough context that the question flow makes sense. An approach that’s often successful is thinking of your survey like a funnel: begin with the broader questions and then move through to more specific questions at the end of the survey.

Leading or Loaded Words Carry Hidden Meaning

 

The language you use when writing your survey should be as clear, crisp and straightforward as possible. The less open to misinterpretation your word choices are, the more effective your data gathering efforts will be. Word choice is important in numerous ways. Avoid complicated terminology or jargon when possible. It’s also important to recognize when a word could be considered leading or loaded. A leading term influences the respondent to answer in a certain way. Loaded words can carry hidden meanings that trigger a different response than what you envisioned. Testing your questions on a few preliminary respondents will help weed out questions that are unclear or biased in some way.

Be Specific About What You Want to Know

 

One common problem that survey writers encounter is using words or phrasing that may not have shared meaning among all respondents. For example, if you ask a person, “Do you work out regularly?” you’re likely to receive a variety of different answers. One interpretation of regularly could mean going to the gym daily; another could believe that playing one game of tennis weekly qualifies. Instead, it’s more effective to ask, “How often do you exercise?” and then provide a range of answers that cover different options: Daily, 4-5 times per week, 2-3 times per week, once a week, less than once per week and so forth. Evaluate each question to make sure that what you’re asking is specific and clear enough to be interpreted by all readers in the same way.

Ensure that Your Answer Sets are MECE

 

It’s critical that answer sets to multiple choice questions be mutually exclusive and comprehensively exhaustive (MECE). Nothing is more frustrating to a survey taker than tackling a survey and finding a question that doesn’t have an answer that fits them. There are two solutions to this. One is to give detailed thought to each of your answer sets, to make sure that you’re accounting for all the variables. Another is to include an option like “Other, please specify” to capture anything you might have missed.

Avoid Double-barreled Questions

 

Double-barreled questions are the ones that inadvertently ask two things, rolled into a single question. For example, a question such as, “Which is the most affordable and most appealing option in this product category?” is really asking two things. The product that’s the most appealing may be quite different from the one that’s most affordable. Often, this problem comes up in hidden ways, so write your questions with this in mind and then evaluate whether each question is asking one specific thing.

Beware Dichotomous Questions

 

A dichotomous question poses a query and then gives respondents two options. For example, true or false and yes or no questions are common variations on this format. Sometimes, researchers use dichotomous questions to help force a choice from respondents on specific issues. But you can run into trouble when you have two options that are not always viewed as mutually exclusive. For example, “Do you consider being Amish to be a religious choice or a lifestyle choice?” Many would argue that it’s both, and such a question is likely to confuse or frustrate respondents.

Invest the time to learn survey writing and development best practices. From how you word your questions to how you build your answer sets, each choice has a dramatic impact on the quality and data outcomes of your survey.

Are you planning to launch your next market research initiative? Contact SSI to learn more about our products and services.