Dr. Pete Discusses Online Research
2011/04/12
Dr. Pete
Dear Dr. Pete:
From what I read, there seems to be an ongoing debate as to where the problems lie in online research. Some say the respondents are bad and others seem to blame the researchers and questionnaire design. What's your opinion?
-Onwan Ha'Nde
Dear Onwan:
I'm on the side of the little guy here. The researcher wrote this junk, hardly the fault of the poor respondent if he can't make heads nor tails of how he is supposed to answer. Sure, there are a few bad apples in the respondent barrel but they are very much in the minority.
Let's put it this way - you write a bad question then all your data is junk. If 5% of your sample contains people who aren't really trying, then 5% of your data is junk. Which would you prefer?
So why don't researchers just get better at questionnaire design, taking this element out of the equation? Then it would all be down to the bad respondent. Two reasons, I think:
- an unwillingness to change: to client, "You know all that stuff you've been paying big bucks for all these years? Well, there's something I need to tell you..."; and
- a whole Dunning-Kruger thing: to self, "I'm good at this, I must be, I get data, I draw charts...".
As long as we continue to accept "unskilled and unaware" as a reasonable description of a market researcher, nothing is going to change. Compulsory PRC, anyone?
-Dr. Pete
Dear Dr. Pete:
I fail to see how anyone could contemplate doing online research. The non-coverage errors alone make it unsuitable for serious study. Surely telephone must remain the pre-eminent gold standard for market research?
-Ma Bell (Mrs.)
Dear Ma:
As you'll know from reading my columns, I can get a little worked up over some things. Fortunately this is not one of them.
Whatever the theory may say, you know we've been dealing with non-coverage on a practical level since forever. We all know of streets where interviewers would rather not venture, types of people you would think twice about approaching in the street, and names you can't pronounce in the phone book. This is all non-coverage and not really an issue online where all we have is an email address. So, we trade-off one sort of non-coverage for another.
But there are non-coverage issues with online, even in a country like the USA with a headline Internet penetration of 72.5%. Get into the older age groups and use of the Internet starts to decline quite dramatically.
What of the "gold standard?" There is a serious and growing non-coverage error in using RDD landline numbers, and this is the growing number of people electing to have only a cell phone. This stands somewhere in the region of 20%. In this instance, the bias is at the other end of the age range, younger people are abandoning fixed-line telephony in favour of mobile.
Does this non-coverage matter? Not always but sometimes. It's our job to know, or at least to have thought about, all the biases associated with each of the data collection methods we can employ on any project.
-Dr. Pete
Dear Dr Pete:
I have a client who wants me to add a whole bunch of exciting and interactive features to his questionnaire. He says that this is what the Internet is all about and that the attention span of people has diminished. I'm not so sure that this won't affect my results.
-Ivor Goldfish
Dear Ivor:
You are so right not to be so sure. Never forget: this is a science and we are doing controlled experiments to uncover opinions.
I've also heard this attention span argument posited as fact and have yet to hear any proof of it. Having received your letter I did a little web-searching to see what I could find. I can tell you, it ain't easy finding academic work on this subject. Lots of "sciency" stuff, lots of stuff from people with axes to grind but nothing much academic.
Here's one that looked vaguely academic:
A. H. Johnstone and F. Percival observed students in over 90 lectures, with twelve different lecturers, recording breaks in student attention. They identified a general pattern: After three to five minutes of "settling down" at the start of class, one study found that "the next lapse of attention usually occurred some 10 to 18 minutes later, and as the lecture proceeded the attention span became shorter and often fell to three or four minutes towards the end of a standard lecture.
"Kids today - no attention span at all.*
One thing is abundantly clear from reading around this subject and that's the difference between attention span"" meaning ""how long you can concentrate on the task at hand"" and ""attention span"" meaning ""how long you can be bothered to pay attention to this."" Your client (I think) is concerned about the second of these and blames it on the first."
I see these as two entirely different concepts. Personally, I can easily understand how, in a world of fragmented media and so much disinformation on the web, people quickly skim anything given them to see if it is worth their time. The important question to me is, once they decide to settle down and read it, can they concentrate on it? This I cannot find the answer to (yet).
So, how do we researchers hope to compete against other things that vie for the respondent's attention? Simply by being more interesting and more engaging in what we do. Respondents aren't forced to our questionnaires; they come of their own free will. All we need to do is live up to expectations. Allow them to share their opinions safe in the knowledge that the information given will affect decisions on products or services or whatever. Present them with a bunch of dumb questions, badly written, and they may start to think you are not serious in your endeavour.
I would say if you feel you need to dress up your question with "whizz bang" then there is probably something wrong with your question, not with your respondent.
-Dr. Pete
P.S. If you think paying 50c gives you the right to enforce attention, think again.
P.S.S. *The study was conducted in 1976.
