Dr. Pete on Data Quality
Dear Dr. Pete...
Dear Dr. Pete:
I don't want to use online research because I've read lots of papers and articles talking about data quality - we never get this in telephone research.
Alex G. Bell
Dr. Pete replies...
Dear Alex:
I guess you've never worked in a telephone unit and seen how concerned they are about data quality.
Not giving honest and thoughtful answers is what leads to poor quality data. Let's take a moment to think about motivations to cheat. Who in the telephone unit has the motive, means, and opportunity to cheat? The interviewers, of course. They know the quick way through a questionnaire, which quota cells are full, how to qualify for the survey, etc., and they are in charge of keying in the answers given. That's why the quality spotlight is pointed at the interviewer with random "listening in," call backs to respondents, etc. The same is true of face-to-face interviewing. Of course, this is all private data. No fieldwork company is going to wash its dirty laundry in public when it finds the odd cheat amongst its interviewers.
Now let's look at an online situation. Who has the motive, means, and opportunity to cheat? Respondents, of course. But in this scenario there is no way of doing the simple checks you use in telephone research. Panel companies have to be more creative in the ways they catch people trying to "game the system."
So why do they go out and tell the world what they are doing? Simple really; by acknowledging the problem and providing a solution, the panel company has a USP that they can use in their marketing. This is your good Doctor with his cynic's hat on, of course. Actually, panel companies all have a vested interest in the health of the overall market. If no one had any faith in online research, then they would all be out of business.
Last thought - if you have one cheat in your online survey, then you will have one bit of bad data; if you write a poor questionnaire, it will all be rubbish.
Dr. Pete
Dear Dr. Pete:
I recently did a survey online; it's one we have done by telephone for years. The first thing we noticed about the online survey was that the number of "don't know" responses has gone up. Is this an indication of poor quality?
Frank A. Bagnale
Dr. Pete replies...
Dear Frank:
Ever wanted to impress someone, even a stranger, with your knowledge? Or at least not wanted to admit your ignorance?
This is what happens in the telephone survey:
Someone asks you a question you don't know the answer to,
Then they read out a list of possible answers.
You don't wish to appear foolish, so
You pick one -
Job done.
People give the weirdest answers to even the simplest questions. Take these from "Family Fortunes" (or "Family Feud" as it's called in the U.S.):
Something with a hole in it? A window
Something people might be allergic to? Skiing
Something associated with pigs? The police
A non-living object with legs? A plant
A domestic animal? A leopard
Online, no one knows or cares whether or not you know something. Respondents, therefore, feel more comfortable admitting they don't know.
Dr. Pete
Dear Dr. Pete
My data just doesn't look right to me. My deadline is looming and I'm getting worried. I don't know where to start. Can you help me?
Nervous Wreck
Dear Nervous:
Well, you haven't given me much to go on so here's Dr. Pete's Handy Guide to Data Problems - a cut-out-and-keep guide to investigating problem data.
Step 1 - Check your tables against the actual original data collected.
Believe it or not, just because a table adds up doesn't mean it is right. It's incredibly easy for the person doing your tables to specify the wrong column. It'll still produce a table, just not the one you were expecting. Any processing of data is liable to human error so it is vital you check against the original data. Remember, this may not necessarily be the data set your DP ended up using.
Chance of error finding: very good; fixability rating: good.
Step 2 - Check your CATI/CAPI/CAWI programming.
I know you signed this off before it went into field but you may have been having a bad day.
Chance of finding error: reasonably good; fixability rating: not good
Step 3 - Review your questionnaire or get someone else to do it (preferably the office pedant).
Is it really fit for the purpose? Is there any ambiguity in any of the questions? A good trick here is to try to answer the questionnaire as if you were totally dumb and couldn't answer anything - would you get stuck somewhere? Are there questions where you would be forced to make something up to move on?
Chance of finding error: unfortunately, quite good; fixability rating: unfortunately, not good at all
Step 4 - Check your sample.
Is it balanced to the population? If not, try weighting (you never know...). Could it be biased in some way? Check your introduction - does it give too much away?
Chance of finding error: slim; fixability rating: not good at all (unless it's a weighting thing)
If all the above come to nothing...
Step 5a - Go off sick.
For a long time. If the client is a big one, think about an alternative career.
Chance of working: this is a one time only solution, use with caution.
or
Step 5b - Call in a statistician.
He or she may be able to convince your client it is all down to sampling error without making it sound like "error" is a bad word.
Chance of working: slim to none, but at least now it's someone else's problem.
Best of luck!
Dr. Pete
Question or comment? email Dr. Pete at Pete_Cape@surveysampling.com.
