← All insights

Non-response as data

Every survey instrument produces two data sets. One is the substantive variables (age, education, caste, monthly income, fertility history, land ownership, opinion on the state of the ration shop) that the analysis will summarise, cross-tabulate and model. The other is the response pattern: who answered which questions, who declined, who said "don't know," who claimed not to know something they clearly did know, and at which point in the interview the respondent's willingness dropped. Reports present the first and usually discard the second.

The response pattern records what the respondent was comfortable telling the enumerator who asked, with the household members who were present at the time. Studying it as data changes what a survey can tell you.

The mechanics are straightforward. Every questionnaire records item-level response codes: response given, response refused, response not applicable, don't know, and skip pattern completed. Most analyses collapse these into "response" or "missing" and impute the missing values. A more informative treatment keeps the non-response codes as a variable of their own and asks three questions of it. Which items have elevated refusal rates? Which subgroups refuse those items disproportionately? Where in the questionnaire does the refusal rate rise, and does the rise coincide with a change of topic?

We have seen four patterns often enough to write them down.

Refusal varies by item, and it follows topic sensitivity. Income, land ownership, household violence, sexual and reproductive health, caste of neighbours and opinions on named political figures produce refusal rates well above the survey average, while age and education produce rates near zero. The relationship between refusal rate and topic sensitivity is monotone. A survey with a refusal rate on income under three per cent has either an unusually cooperative sample or income data confabulated by respondents who did not want to say "refused" out loud. It is usually the second.

Refusal also has a subgroup structure. Women in patriarchal households refuse political and financial-decision items more often when male household members are present, and less often when the interview is conducted privately. Caste-sensitive items produce different refusal patterns among upper-caste and lower-caste respondents even when the item is neutrally phrased. When the question asks whether the respondent's family has faced discrimination, the refusal pattern by caste is a finding in itself, and often a more reliable one than the substantive answers.

Refusal accumulates within an interview. If a respondent refuses an item at question fifteen, the probability of refusal on later sensitive items in the same interview rises. Interviewers who are good at recovering rapport slow this drift, and interviewers who press on through the script speed it up. Correlating refusal across items within an interview is a check on interviewer effects. Most analyses do not run it, though it is easy to do.

The don't-know code behaves differently from the refusal code. Respondents use "don't know" when they are unwilling to commit publicly to a position, or unsure whether the interview is confidential enough for a specific answer. Its rate rises on items where being seen to hold a position carries a political or social cost. Coding "don't know" as a refusal merges two different behaviours, and treating it as random measurement error understates how patterned it is.

Two things follow for analysis.

The first is to report response and refusal rates by item, subgroup and interview position. The table takes little analytic work, and it changes how the substantive results should be read. On sensitive topics a published table of refusal rates by item is often more informative than the matching table of substantive responses. Reviewers who take non-response seriously read it first.

The second is to model the response indicator alongside the substantive variable. If the outcome is household violence in the last year, the analysis gains from a joint model of the substantive response and the refusal indicator. A respondent who refused the item is not analytically equivalent to one who reported no violence, and the two should not be pooled in the summary statistic. Multiple imputation with an indicator for missingness is the standard reference for this approach. It is under-used in the applied literature we read.

A note on ethics. Non-response is the respondent's exercise of a right, and analysing it should not become a way of extracting information the respondent chose to withhold. Our analytic interest is in the pattern of refusal across items and subgroups. Inferring what any individual would have said falls outside it. A study that imputes its way past non-response ignores what the respondent was communicating by refusing. Read as data, the refusal tells you about the survey's relationship to the population.

The best survey enumerator we ever worked with told us, at the end of a long day in a difficult district, that the most useful thing she had learned was to write down which items respondents paused on before answering. The pause was almost as informative as the refusal, she said, and she was right. Timing data from digital collection platforms such as SurveyCTO, ODK and KoboToolbox records this at almost no marginal cost. Most analyses discard it, even though the pattern of pauses at sensitive items is data in its own right and often turns out to be the main result.

Useful references: Little and Rubin's Statistical Analysis with Missing Data remains the standard reference on non-response modelling; Groves et al.'s Survey Methodology covers the field-level dynamics; and van Buuren's Flexible Imputation of Missing Data is the practical treatment.