← All insights

Attrition isn't random: the households panel surveys lose

A panel survey follows the same households over time. You can see how a livelihood shock spreads through a household, how a programme effect builds, and how vulnerability changes over a season or a decade. Panels also carry a bias that cross-sectional surveys do not have, because attrition between rounds is rarely random.

Households leave panels for reasons tied to the outcomes the survey is measuring. Some migrate after a drought, a flood or a collapse in wages. Some dissolve under household conflict, which is often itself a result of the stress the panel is trying to capture. Illness or death changes who lives where, and the household can no longer be found. Others withdraw when programme contact falls away and the survey matters less to them. The households whose trajectories the panel was designed to follow end up systematically over-represented among the missing.

The standard corrections deal with part of this. Inverse probability weighting on baseline observables (education of the household head, an asset index, district fixed effects, the baseline value of the outcome) corrects for the attrition that is associated with variables you observed at baseline, and it is worth doing. It does nothing about an unobserved shock that drove both attrition and the outcome of interest. That is the larger source of bias, and weighting leaves it in place.

In a livelihoods panel in a flood-affected eastern district, attrition between baseline and the second follow-up reached 22%, which is not unusually high for the context. The households lost had baseline characteristics that already predicted higher vulnerability: a lower asset index, more migrant labour, fewer adult earners. IPW on those characteristics narrowed the gap between the balanced panel and the original baseline distribution. It did not close it. Diagnostic checks comparing balanced-panel respondents with administrative records of NREGA participation in the same villages suggested that the corrected sample still under-represented the most-affected households. The published estimate of programme effect was in all likelihood an upper bound, and the report did not say so in those words.

Heckman selection correction can in principle handle non-random attrition, under a stronger assumption: at least one variable affects the probability of being observed and has no effect on the outcome. In development settings the search for such an exclusion restriction usually fails. Whatever predicts whether you can find a household again (mobile phone access, residential stability, neighbour cooperation) tends also to predict the welfare and livelihood outcomes the panel measures. Analysts sometimes apply the correction anyway with an implausible restriction, on the grounds that something is better than nothing. With such a restriction the corrected estimate is often worse than the uncorrected one.

Tracing protocols are routine work, and they move attrition more than any statistical correction can. Mobile-first tracing (several phone numbers collected at baseline, a monthly check-in SMS, follow-up calls before each round) reduces silent attrition meaningfully. Neighbour informants help, especially in tracking migrants, and a well-designed protocol collects contact details for two relatives outside the household at baseline. Administrative records such as NREGA muster rolls, PDS dealer lists and school enrolment can identify households that have moved within the district and lost their original locator.

Attrition reporting also has a political economy. Panels with low attrition are easier to publish, win contract extensions and produce headline numbers, while high attrition is uncomfortable to discuss in a funder meeting. Teams therefore have an incentive to report the balanced subsample as the main analytic sample, sometimes without making the selection step visible. The headline number is then conditional on the household still being observable, and the original research question was meant to hold without that condition.

A better reporting standard would set a threshold (we use 15% as a working rule, and others use higher ones) and treat attrition above it as a problem that changes how the analysis is framed. Statistical correction alone does not answer it. Above the threshold, the report should describe what is known about the households that left, from administrative cross-references, neighbour reports and partial information from incomplete follow-ups. It should report the analysis on the full baseline sample, with those households assigned a range of plausible outcomes, and present the headline estimate as conditional on observability. This takes more work than it sounds. A panel reported this way supports credible inference, while one reported only on its balanced subsample gives a clean number that may be wrong.

Panels that report low attrition often get there by selecting on stability at baseline, for instance by sampling only households with at least one earning member who has lived in the village for five years. The low attrition then follows from having pre-selected the stable part of the population, and the selection problem has simply moved to an earlier stage. Published methodology rarely flags this. The diagnostic is to compare the composition of the eligible sampling frame with the achieved baseline sample. If a report does not present that comparison, a reader cannot detect the substitution.

Useful references: Wooldridge's Econometric Analysis of Cross-Section and Panel Data for the formal treatment of attrition; the IPA Goldilocks tracking working paper for practical tracing protocols; and the India Human Development Survey documentation, which is unusually transparent about attrition and tracing across rounds.