Dr. Pete on Data Quality
12/04/2011
Dr. Pete
Dear Dr. Pete:
My client says he will award the contract to the agency that can provide the best quality - but then doesn't say what the quality criteria are. How should I proceed?
-W. E. Deeming
Dear W. E.:
With caution young man. Quality is often said to be defined by the customer. That is to say a quality product is one that is fit for the customer's purpose. At the very general level of course this is a good working definition for quality within market research. A proposed research solution that meets the customers stated need is a better quality proposal than one that doesn't. But how far can this "keep the customer satisfied" go? Once we get past the proposal stage we are in the business of providing Quality Data, and now our Quality objectives may not always be 100% aligned with those of the client.
It may come as a surprise to you, gentle reader, that not everyone who commissions a piece of market research is interested in an unbridled search for truth. The research may be required to back up someone's pet theory, and if the research does not stack up I can guarantee that it is not the pet theory that's going to be abandoned. Alternatively bonuses and commissions may depend on the research outcome and where money is involved there is always the incentive to cheat a little.
What then is the lowly researcher to do, client wants answer X and wants you to guarantee to get it. You know exactly how to get answer X, but you also know it is not the truth. This is where your ethics come in. If you don't have an industry Code of Ethics then it has to come down to your own moral code. Maybe you can sleep at night knowing you willingly took part in a deception but, don't forget, it also makes me and everyone else in the industry look bad. And that, paraphrasing Winston Churchill, is something up with which I cannot put.
-Dr. Pete
Dear Dr. Pete:
I've a nagging feeling that there is something not quite right about my data, but I can't put my finger on it. The frequencies look sort of okay but I just don't know. Can you help?
-Lee Littletime
Dear Lee:
Indeed I can, at least I can if you have the right sort of questionnaire. This possible problem with your data is either going to come from the people in your sample not answering your questionnaire properly or without the right level of care and attention, or the questionnaire itself is not asking the right question or the question in the right way.
Let's take the second point first. It's like having a thermometer that doesn't measure the temperature correctly, but does so consistently. You can take lots of readings and get a general idea of changes in temperature but, when faced with the acid-test question; "what is the temperature now" your instrument is going to fail to tell the truth. The difference between a market research questionnaire and a thermometer of course is that no one actually knows what the truth of the answers to the questions are! What you get left with is that nagging doubt that you have about the veracity of the data. All you can do is get some alternative opinions on your questionnaire. Does it really ask what you think it asks?
The first point is actually easier to deal with. In most questionnaires there will be certain questions that sort of "hang together" (as my professor would say). If you answer Q1 in a certain way you will probably answer Q3 in a certain way also. What the academics would say (and please bear with me here) is that Q1 and Q3 are a set of items measuring a single, uni-dimensional latent construct, there I've said it, let's move on now. Now if some or all of your respondents are not really paying attention to the questions they may well answer Q1 and Q3 inconsistently. When you look at your frequencies you may not see this inconsistency, it would be revealed only by cross-tabulating the relevant questions or by calculating a special statistic - Cronbach's alpha.
I said at the start that my ability to help depends on the sort of questionnaire you have. We commercial market researchers don't always think about things like 'validity' and 'reliability' as much as we perhaps ought to. Since we therefore don't plan in questions designed to test for validity or reliability we have no recourse when we get the nagging doubt that that there is something wrong.
Here's today's take-away: if you are doing a self-completion survey ask yourself how you are going to know if the person filling out the survey is telling you the truth. If the answer is "I dunno..." then think again about your questionnaire.
-Dr. Pete
Dear Dr. Pete:
It seems pretty obvious to me, and makes perfect common sense, that if you have a data quality issue then it has to be to do with the sampling. Surely you must agree?
-F. Boas
Dear F:
Did you know that in the Inuit language there are over 100 different words for snow where English has just one? Makes perfect common sense when you think about it, they spend all their time surrounded by it and need to communicate the fine differences in snow types to their friends. All well and good except for one little thing - there are probably no more words for snow in Inuit languages than we have (think slush, sleet, hail, blizzard, powder...). This is an urban myth, and one of the best ones around. Myths are often validated by a recourse to a notion of "common sense".
But we are scientists (when we are not being artists or craftsmen that is) and we are convinced only by proof. I've said this before and I'll no doubt say it again. Market Research is simply about asking the right people the right questions and understanding their answers. Get any part of this wrong and you are going to have a data quality issue.
Cheers!
-Dr. Pete
