The effectiveness of an educational policy for improving vaccine uptake: Persistence and heterogeneity of effects

Enormous progress has been made in reducing child mortality and disability over the last two decades in low- and middle-income countries. Childhood vaccinations have played an important part in this success story. They represent one of the most cost-effective health technologies, in that they prevent mortality and disability at relatively low cost. Despite the well-documented evidence and consistent investment in national immunisation programmes, WHO estimates that globally 23 million infants were not fully vaccinated in 2020. In Uttar Pradesh, the most populous and poorest state in India and the setting for this study, recent estimates show that 70% of children aged 12 to 23 months are fully vaccinated. While this represents a marked improvement over the past five years, it is clear widespread availability of free immunisation services in public facilities is not sufficient to guarantee high coverage in the population. An experimental evaluation of a door-to-door information intervention to educate mothers on the benefits of child vaccination, reported large positive effects on vaccine uptake in the short-term. The intervention provided the mothers of unvaccinated or incompletely vaccinated children aged 0 to 36 months with health information on the benefits of vaccination. We extend this evaluation, using 23 months follow-up data for mothers of children included in the original experiment, to address two outstanding policy questions.

First, are the effects of the intervention sustained over time? It may be the case that the initial effects of the intervention are attenuated over time, if the intervention merely brings forward vaccinations that would have happened anyway. Under such a scenario, any health benefits of vaccination would be temporary, undermining initial estimates of the cost-effectiveness of the intervention. We also consider whether the intervention led to a sustained change in parental behaviour, be examining effects on the vaccination of particpants' children who were not born at the time of intervention. Such persistence of effects would increase the cost-effectiveness of an intervention.

Second, who benefits from the intervention? Information on heterogenous treatment effects has various uses. It can inform policymakers as to who should be targeted by the intervention to maximise take up of immunisation. It can speak to the question of equity and can potentially shed light on the mechanisms through which the intervention worked. Finally, it can offer policymakers insights on what other interventions may be needed in tandem with demand-side strategies.

The main outcomes were the proportions of children who received (a) diphtheria–pertussis–tetanus (DPT3), or (b) measles vaccinations. We reported whether the overall effects of the initial study were sustained over 23 months, and heterogeneity of the individual-level effects. Attrition was low; 93% of the 722 mother-child dyads who were randomised completed follow-up at 23 months, and baseline characteristics amongst this subsample remained well balanced between the randomised groups. We estimated effects with Causal Forests' (CF), an ensemble Machine Learning methodt hat can predict HTEs according to observable characteristics by searching over high-dimensional functions of covariates rather than requiring the a priori specification of outcome models.

We find that the large short-term effects of the information intervention were sustained at 23 months, with considerable heterogeneity in the individual level-effect estimates. While we detect positive effects for siblings these are not statistically significant, perhaps reflecting reduced statistical power as the analysis only includes 293 siblings. Thus we cannot be confident that the intervention led to lasting behavioural change. This evidence provides important context for the original analysis and can help target educational initiatives to improve vaccine uptake in low-income settings.

Vaccination

Randomised Controlled Trial

Heterogeneous Treatment Effects

Machine Learning

Causal Forest