Dr. Pete on Cross Cultural Research
2011/4/12
Dr. Pete
Dear Dr. Pete:
I've been commissioned to conduct some multi-national research. It seemed pretty easy, just a bunch of attitude batteries using 1 to 10 scales, but now my client is asking if the results are a true reflection of real differences between cultures or just the way in which different cultures answer questions.
Help!
-SS Stevens
Dear SS:
Your client is to be congratulated for asking the $64,000 question and you now know why international research is so difficult. Yes, it's really easy to conduct this stuff, without even having to leave the comfort of your office, but what does it all mean?
We researchers can be a funny lot. We recognise that there are differences in behaviour between people in terms of their behaviour and attitudes (after all, that's what we are trying to measure), but we ignore the uncomfortable fact that different people might fill in our questionnaires differently. I guess much of this stems from not having a point of reference. We hope and pray that there is no "cultural effect" and that one person's score of 4 is the same as the next person's score of 4. If we really knew what a 4 meant for every person personally we could weight the responses.
Of course, this is all nonsense. Real people in their real lives don't think in terms of scores "out of ten." When you decide to buy one brand rather than another, you just do it. You don't total up a bunch of scores on various attributes and weight them according to your personal attribute importance; you just make your choice. Imagine a crazy scenario where all the people who use brand A have a personal top scoring of 8 - they don't believe that perfection can be achieved. Alongside them are all the people who use brand B. They are the opposite, they believe every company is trying its best and so all companies must be scored at least an 8. The scores for brand A will not exceed 8 and the scores for brand B will be at least 8. Does this make brand B better than brand A?
This in a nutshell is the problem facing researchers doing international research and using scales. Now I know you're all asking yourselves, "Is Dr. Pete going to reveal the calibration or weighting necessary to correct this cultural difference?" Unfortunately not. If I knew the easy answer to that, I would be a rich man. The extent of the "problem" depends on the nature of the research and the subject matter. Certainly you can see if there is a cultural effect by plotting the Top Box scores for unrelated questions Country A against Country B. If you see a pattern that all of Country B's Top Box scores are lower than Country A's Top Box scores, then you might assume that a cultural effect exists rather than a real phenomenon. This just might be enough to head your client off at the pass. It is even possible to calibrate from this type of analysis. Another alternative is to include some culturally neutral statements to try to estimate a true score equivalence on the given scale.
It's not just market researchers who struggle with this stuff. Psychologists, management scientists, and health researchers all have to cope with the same problem. Just type "calibrating scales in cross cultural studies" into Google for some interesting academic reading.
I hope this helps.
-Dr. Pete
Dear Dr. Pete:
I've been commissioned to conduct some multinational research. It seemed pretty easy, just a bunch of attitude batteries with agree/disagree scales. Now my client is asking if the results are a true reflection of real differences between cultures or just the way in which different cultures answer questions.
Help!
-LL Thurstone
Dear LL:
As we say in England, this is like catching a bus - you wait forever, and then two arrive at the same time! You've got the same problem as SS - but even worse.
SS's problem was all about the use of numbers, but at least numbers are pretty much universal. You have the added complication of language and translation. Whilst all people in the same culture are likely to pretty much agree on an understanding of what "Completely Agree" means, there is lots of room to interpret what "Somewhat Agree" might mean. Imagine then the disparity across cultures on what "Somewhat Agree" might mean - we can call this the "SS effect."
Now let's add in the complication of translating "Somewhat Agree" into other languages. Suppose the language doesn't have a direct translation of "somewhat." Then should they use "nearly?" "partly?" "almost?" Now we have a scale that has a different mental distance between the three points neither/nor, somewhat, and completely agree. And we all know what we get from different questions - that's right, different answers.
I don't have an easy answer for this one either, but you might try looking at the summaries of agreeing and disagreeing. The direction of opinions may be the same across the cultures even if the overall scoring isn't. Also, you can try that cross-plotting-of-unrelated-questions trick. After that, you're going to need to start thinking about acquiescence bias, so I hope you have a pair of opposite statements in your survey to see if this is an issue.
Good luck!
-Dr. Pete
