Calculating Incidence on the Question Level

By Keith Phillips, Research Methodologist

When we talk about incidence as a sample supplier we are often referring to the survey incidence.

Survey Incidence = [the number of completes/(the number of completes + screenouts.)].

This gives us information as to what proportion of participants will qualify and how much we should charge per complete for the study.

However, incidence broadly means the frequency of something occurring within the population (like incidence of smart phone ownership in the US).  When we are calculating incidence in this way (call it the question incidence), we should be analyzing everyone asked the question and not worrying about excluding drop outs or quota fulls within our data.  In fact, depending on what else is happening within your survey, excluding responses from an incidence calculation could bias your result.

Survey Incidence

For example, if you quota’d for usage segments (different levels of TV viewership) and those quotas came after a question that screened participants out of the study (must watch some TV per week), you would be under calculating the incidence of TV viewers by using only the completes and the screen outs within your calculation.  Why?  Because everyone who doesn’t meet your screening criteria is counted in your incidence calculation (screen outs) and those who meet your screening criteria are counting only when their quota is open.  You are systematically removing TV viewers by not including the quota fulls in this example.

Here is the same example of the “question incidence” with some hypothetical data:

2000 starts get asked a question about TV viewership (for this example we assume that the starts are representative of the population being sampled).

Of these 2000 starts, 1000 qualify for the study as TV viewers and 1000 screen out.  (50% incidence of TV viewers)

500 of those qualified went on and received a quota full due to their TV viewing segment already being full.

The remaining 500 that qualified went on to complete the study.

If you were to include only the completes (500) and the screen outs (1000) in your calculation of TV viewership you would conclude that only one third of participants are TV viewers (500/1500), but as we saw earlier, the true incidence is 50% (1000 TV viewers out of 2000 starts).

So make sure when you calculate incidence on the question level you are using all of the available sample and not mirroring the calculation we use on the sample end to determine the efficiency of the study.