When a survey reveals to you that non-citizens voted in US congressional Election Studies, what are you to do? Do you report it as fact and get abused all around or have a hard think about it and bury it?
This is the choice that no longer faces Jesse Richman, Gulshan Chatta and David Earnest of Old Dominion University. They wrote a paper on the subject that was published in the journal Electoral Studies in December 2014. There it may have languished, but it has resurfaced after having been quoted by President Trump as a contributing reason why he did not win the popular vote. At the same time, it is being roundly condemned by Democrats as fundamentally flawed.
Without going too much into this particular case, the issues it raises are pertinent to our everyday lives in market research. A big part of the issue is simply measurement error. We tend to ignore this as noise in the data and sometimes, in fact most times, it is invisible to the naked eye. Once noticed however its size, impact and implications for the validity of the rest of the data can be blown out of proportion.
Measurement error covers a number of sins from confusion, through ignorance to carelessness and on to dishonesty. For example, the most innocent of these in online research, is when the participant clicks “yes” when they genuinely meant to click “no.” The measurement errors work both ways in that some people click “no” when they meant to click “yes.”
What happens as a result depends on the nature of the survey. Sometimes the survey routing will take the participant to a place where they cannot answer the questions and it is obvious to them that they must have made a mistake. In this case, the participant can back up and correct themselves. This is unless, of course, you disable the back button. If you do, then they have a difficult choice to make; continue ‘lying’ and hope to get back on track, or crash out of the survey.
Often however it is not obvious to the participant that they have made a mistake, it just shows up to the researcher in the cross-tabs. They may clean up this data or not, as they desire, but they would rarely highlight this oddity as a serious finding.
Where life does start to become difficult is when you are dealing with low-incidence subjects and this is the population of interest. This is where Bayes Theorem comes into play and where many get hung up on potential data quality issues when only measurement error has occurred.
The rate of miss-clicking, for any given question, is fixed and constant and random. We’ve experimented with trying to find out how big an issue it is by simply collecting some data and feeding the answers we gave back to the participant and then asking if that was what they intended to say.
Some of the worst measurement errors we found were in collecting dates of birth. This of course is three separate bits of data; Day, Month and Year. In totality, without any error checking in place, we found 5% entered their date of birth incorrectly, 2.5% got the day wrong, and 2.2% entered the year wrong. “Only” 1.1% got the month wrong. These are consistent with the error rates we found when checking other numerical data input which we also found to be 2%.For single-coded questions the error rates are much lower. We found just under a 0.5% error rate on a single coded question.
Just how much damage could that do to your data? Here’s where Bayes Theorem comes in. Let’s say that the error rate for the question in a survey is 0.5%. Let’s also say that the real answer in the population is 1%.This means that there really is a 99% chance that anyone you question should answer “no” to your question. Given the error rate we can say that; of those who should click “no – I am not this”, 0.5% will make an error and will click “yes – I am this.”
Since only 1% of the population should properly answer “yes” (and 99.5% of them will do so correctly) you can see that one in three of the “yes” responses you get will come to you in error.
This can rapidly add up to a large number of people. If you only start with 500 people being asked the question then you will get a total of seven people answering “no.” This is made up of five people answering correctly and two who should have said “no” but said “yes” in error. Your overall answer is a little over 1% which fits your prior expectation and is within your margin of error.
Generally seven people is not a big enough base to warrant much further examination and this then remains noise in the data that may or may not be cleaned out – it will certainly be ignored.
Now see what happens when the sample size is increased dramatically. What if we started with 55,000 participants, looking for the low-incidence group to be a reasonable base size. Now we have 820 people who state “yes” to the question. This of course is still ~1% of the total sample. Of these, 272 should really have said “no” but clicked “yes” in error. The percentages all remain the same but the base size starts to become analyzable and 1:3 of the participants are there in error.
Therein lies the danger when searching for low-incidence targets. Without a checking question and either a straight disqualification or an opportunity to revisit the earlier question then the danger is real that a substantial number of participants are not who you think they are.
If you substitute “citizenship” and “voting” into the scenario above, you start to see the problem that Richman et al potentially face. Non-citizenship is the low-incidence number and where the error may occur. Voting is a high incidence activity among citizens and also therefore among those accidentally classifying themselves as non-citizens.
So how does this apply from a data quality perspective? The issue arises when the researcher begins to confuse error with inattention or worse, fraud. As we have just demonstrated, quite large numbers of people can be “caught in a lie” just by asking the same question twice. Researchers do often point the finger at such behaviour and wonder if the rest of their data is valid or not. Our standard response is to go through the Bayesian statistics based on a 0.5% error rate and show how the numbers of people arriving at their survey in error can occur.
The second point we make is that these people are not behaving like fraudsters. Once in the low-incidence survey they disqualify themselves! In fact we recommend repeating the same qualification question twice in order to eliminate those who have arrived in error. Far from being “caught in a lie,” this repeated question saves them from having to lie to complete the survey they have entered.
For those interested in reading the full paper by Richman et al it can be found here.
A further, more detailed explication of the impact of measurement error on that study can be found here.