Survey Frustrations: Setting the Right Expectation

By Keith Phillips, Sr. Methodologist

Have you ever shown up at a doctor’s office and then waited for 45 minutes before you they could see you? Or how about making a reservation at a restaurant and then having to wait 30 minutes for your table to become available? It’s pretty frustrating and it is even more frustrating if you are trying to do this within a limited window of time. I remember walking out of a doctor’s appointment after waiting for two hours because I had to get to a work-related meeting. Needless to say, I was pretty angry with the doctor.

Recently, I had a similar experience with a survey. I was making a transaction online, when I was asked to take a short five-minute survey after my transaction about my experience. Those are not my words, I was specifically asked to take a “short five-minute survey.” Working in the market research industry, I was more than happy to oblige. However, ten minutes into the survey, while answering a battery of attributes with no end in sight, I realized that the questionnaire was quite a bit longer than had originally been stated. This was pretty frustrating.

As I made that realization, I suddenly was transformed from a market researcher to a normal survey participant.  I was ready to abort ship. I wanted to drop out of the survey, or at least quickly finish the remaining portion. I mean I’ve answered all of the relevant important questions, right?  Of course, my intimate knowledge of how market research data is processed and my desire to provide quality data for any colleague of mine doing the analysis kept me on track. I couldn’t help but think that a regular survey participant may not have been as forgiving.

It’s all about setting expectations. When people are told they have to wait in line to get onto a ride for one hour at an amusement park and then they get on in 30 minutes, it was a short wait. When you have to wait for fifteen minutes to pay for your groceries and you are only behind one customer, it’s a long wait.

It’s similar with a survey. You can’t tell someone it’s going to be a five minute survey and then have a twelve minute survey experience. That is going to be very frustrating, which could lead to more drops, poorer data quality and perhaps a negative disposition towards surveys in general.

Of course, part of the problem with setting expectations is actually knowing how long a survey is going to take. This is especially problematic when you consider that many people will have a different survey experience, with some getting more questions than others. Additionally, everyone works at their own pace.  For this reason, it’s important to have a conservative estimate for how long the survey may take and then say the survey should take less than this.

It’s also important to consider that as the survey length increases, the group of people who will have that particular amount of time free in their schedule will decrease. This should not move us to underestimate the length. Instead, we want to be upfront with the amount of time needed so that we do not have people rushing at the end of the survey, potentially impacting data quality.

Progress bars can be a good way for survey takers to see how much of the survey is remaining and ultimately estimate how long a survey will take. However, you have to make sure the progress bar is accurate. In some instances, participants will see large jumps in the progress bar when they skip sections of the survey and in other instances, participants are in a battery of questions that makes it appear the progress bar is not moving at all, giving the illusion that the survey is much longer than it actually is. You will need to test if your particular survey works with a progress bar before adding one.

Finally, not mentioning the survey length at all would be a mistake. Remember, nobody wants to be sitting in a doctor’s office with no sense of when they will be called. Setting the expectation is a must.