Dr. Pete on Community-Based Research
12.04.2011
Dr. Pete
Dear Dr. Pete:
What do you think about these new-style community-based research projects? Going to be the death of quant research, don't you think?
-F. Acebook
Dear F:
Au contraire mon petit, I think quite the opposite. This could be the making of quantitative research. It's probably our friends in qualitative that ought to be worried. Let's think a bit. You get a reasonably large group of people in an enclosed space and open up a conversation with them. They chat about this for a bit then one finds something else interesting to say and they discuss that for a bit. You bring them back on subject or get them to chat about something you want them to talk about. At no time are all of them chatting, maybe 20 or so on any one thread the whole time.
Now let's compare this to a focus group. You get a group of people together in a room and open up a conversation with them. They chat about this for a bit then one finds something else interesting to say and they discuss that for a bit. You bring them back on subject or get them to chat about something you want them to talk about. You try to get all of them involved. Sounds pretty similar, n'est ce pas? Only thing that's missing is the discussion that starts "imagine this television was a wild animal..." (To all qualis: you know I'm only kidding.) I would say that traditional qualitative was under much more threat than quant. I'm not talking jobs here, these communities take a lot of monitoring and moderating, but I am talking about having to learn a whole new skill set, and having to let go of some much loved techniques.
For quants of course there is the opportunity to take the community outside to a formal questionnaire setting. For the most part this is going to jar horribly with what they have been doing up until then. Inside the community, it's all been sweetness and light with gentle probing questions and lots of positive feedback for their efforts. Now the gloves will be off, the spotlights turned on, and really difficult questions asked in a completely non-conversational way with perhaps, at most, a little "thank you" at the end.
This is where quantitative is committing a long, drawn out, semi-public suicide attempt. In the race to be faster and cheaper, we've lost the ability to construct a decent questionnaire that engages the minds of respondents, and asks them to think about the whys and the whats of their own behavior. Here's a little secret - they like this stuff. What they don't like are questions that don't make any sense with answers that don't cover their opinions; and irrelevant nonsense about subjects they have no interest in and have told us they have no interest in. Just once in a while every researcher should play the role of respondent and do one of their own questionnaires. They'd quickly understand the meaning of "cognitive burden." No wonder we call it "administering" the questionnaire... [rant over].
But seriously, you can see the attractiveness of the community-based project for the respondent. They speak in their own voice, are listened (and responded) to, and get to speak up about what interests them. For the quant researcher, the positives are less obvious - would you fancy projecting to the population the views of a community who have spent the last two weeks discussing a single issue in great depth?
-Dr. Pete
Dear Dr. Pete:
I know quant research is all about getting numbers but why is it that when I ask a "why do you say that?" type question, I never get any deep responses that I can work with. What I get back all seems so thin.
-S. Hallow
Dear S:
That's because asking a "why" question is, in itself, shallow. It gets you very little to work with unless you can follow it up with probing until you reach a "what was it about?" question. In this respect, quantitative interviewing can seem a little like questioning a teenager about "why" they did what they did. If you get more than a dismissive shrug of the shoulders, you are doing well. Often the very best you might get is, "because it seemed like a good idea at the time."
If you take the time to probe a little, you might get to the truth:
"Why did you do it?"
"Because it seemed like a good idea at the time."
"Oh, that's interesting. Whose idea was it?"
"Dunno. Johnny's I think."
"Ah, Johnny is usually full of good ideas isn't he?"
"No, not really. Most of his ideas are pretty stupid, but the rest of the guys look up to him."
"Hmmm, so what was it about this idea that the others thought was good?"
"Nothing really. We just didn't want to disagree with him. He's stronger than us and a bit of a bully and it didn't seem that anything bad would happen if we did it."
"So let me summarize: You did the deed because you felt pressured by your peer group to accept a course of action that you disagreed with. Your tacit agreement was on the basis that not going along with the action was a greater risk to yourself within the peer group context compared to the relatively minor risk from society at large from doing it. To that extent it seemed like a good idea at the time."
?Can you see parallels with the "why" questions you ask? They elicit thin results because a thin answer suffices to answer the question. Many quant researchers use the "why" question because they have run out of hypotheses to test. If they knew the "whats" that drove the "whys," they would include them in the survey. If you get to the point of having to ask a "why" question in quant then it's time to think about some qualitative research, ask others around you, or pray you get a good set of interviewers on the job.
-Dr. Pete
