Dr. Pete on Respondent Cooperation
12-4-2011
Dr. Pete
Dear Dr. Pete:
What is it with respondents nowadays? I'm shelling out good money for incentives and, still, response rates to my survey are going down and down. I'm not willing to go above 50 cents per 20 minute survey. What do you think I should do?
-Burrhus Peeler
Dear Burrhus:
I definitely think you should not enter the realm of motivational coaching! 50 cents for 20 minutes equates to what, $1.50 an hour. How does that compare to minimum wage? If you think solely in terms of money then this tells me everything I need to know about how you value respondents. Guess you're not going to see the point of this bunch of crazy Swedes:
We believe that the easiest way to change people's
behaviour for the better is by making it fun to do.
We call it The Fun Theory.
Check them out in action (http://tinyurl.com/yclbbf9) and tell me if you think you could have gotten the same result with a 50 cent incentive.
-Dr. Pete
Dear Dr. Pete:
How much is a fair incentive to an online panellist for a 10 minute questionnaire?
-Peirce O'Twine
Dear Peirce:
Blimey, two in a row about incentives... fancy that.
Oddly enough, I was just reading a very nice paper in the IJMR by Larson and Sachau on the effects of incentives on respondents' ratings of products. They took as their start point Equity Theory, which is fair enough. This is not the first time I've seen Equity Theory used in this context but I think they expound it quite well.
As you all know, Equity Theory states that fairness in compensation is not an absolute amount but a perception that the ratio between input and outcome of the receiver and the 'other' is roughly equal. If it isn't, then both parties will deem the relationship to be unfair and will do something to redress the balance - else suffer psychological distress.
What is particularly important in setting up the equation is to understand all the inputs and all the outcomes; this is where Larson and Sachau, in my humble opinion, have done a great job. Their equation has:
|
Panellist |
|
Researcher |
| Inputs: |
Time, Energy, Expertise |
= |
Incentive |
| Outcomes: |
Incentive, Enjoyment |
= |
Valuable information |
So often before I've seen the panellist denominator only contain the word "Incentive" and the Numerator "Time" which made it very difficult to square with panellists who do surveys for nothing. Now with the addition of "Enjoyment" on the panellist side we can understand how the zero incentive situation "works," although this means that the Researcher has no input into the process! I think I would add to the Researcher numerator, "Questionnaire Design Effort." (Remember, you read it here first folks!)
Anyway I digress.
Let's suppose (as Larson and Sachau did) that the panellist feels undercompensated for his or her efforts. Well, the panellist might reduce inputs (refuse, drop out of the survey, or satisfice in the survey) or, and this is the neat bit, reduce the researcher's outcomes by giving not so good information. In the example of a product test, the panellist may rate the product less favourably, undermining the new product development process.
Larson and Sachau did just this in their experimental set-up. They reduced the incentive to a sub-set of the sample to the point that many in the subset stated that it was at an unfair level. To another subset they increased the incentive to a point where many stated it was too much for the task in hand. The third sub-set got precisely what would normally be given for the task, a 10 minute interview.
Response rates and drop outs were equal across the three groups as were interview lengths so the researchers assumed panellist input levels were the same. The survey itself, an NPD on toothpaste, did not differ so we assume enjoyment levels did not differ. But what happened when they looked at the other side of the equation? Well, those undercompensated for their time gave lower ratings for the product, albeit by a modest amount. And what about those overcompensated? Did they give higher product ratings? Actually, they didn't. Apparently it is a an oft seen phenomenon in Equity Theory experiments that people are better able to reconcile overpayment in their own minds, because, as the lady in the L'Oreal ad says, "they're worth it."
So what's the answer to your question? I don't know. It's the amount they expect to get given the task in hand, and this will differ depending on the panel. Mainly this will be measured in time, but not always. Perhaps a tough 10 minute questionnaire requires a higher incentive? One thing to bear in mind is that the effect is small. Much research has pointed to the fact that many people require no incentive at all; many do not even read the invitation and have no idea how much (or how little) is on offer. Fiddling about with incentive levels might be less effective than concentrating on "enjoyment" to get the required researcher outcome. If this means increasing Questionnaire Design Effort, then so be it.
-Dr. Pete
Dear Dr. Pete:
My client and I are arguing. I say his questionnaire is junk; he says just throw some more incentives at the panel to make them do it. Who is right?
-I M Spartacus
Dear I:
You are, my brave little one. I'm so inspired by your fighting talk I feel I should join you on the barricades......but I'm a bit busy.
Your comrade in arms,
-Dr. Pete
P.S. You didn't really use the word junk to his face, did you?
Dear Dr. Pete:
How do you sum up respondent cooperation? It's about us getting people to do what we want, isn't it?
-James Fosmall
Dear James:
No. It's about allowing them to do what they want to do. Think about it. We can try to force people to do what we want but, like the man said, "Might does not make right, it only makes history." Coming from the man who brought us, "Age does not diminish the extreme disappointment of having a scoop of ice cream fall from the cone," I'm buying his worldview.
Toodle pip,
-Dr. Pete
