Using Causal Inference to Inform Survey Administration

A common question asked in a survey research institute is, "what impact does an increase in survey incentive amount have on survey completion rates?" Many decisions concerning increasing completion rates are based on analyzing past survey designs and resulting responses. An analyst typically gathers as many covariates as possible, adds them to a regression model, and interprets the beta coefficient of incentives as a causal treatment effect. However, this approach is incorrect. Ignoring the survey design and how records are kept in a database can lead to selection biases and spurious associations. This poster aims to show how inverse probability weights (IPW) can inform survey administrators how much incentive rewards should be used to increase completion rates for a specific survey. IPWs requires knowledge of the causal relationships between variables to recover causal effects. A directed acyclic graph (DAG) can be used to represent the causal relationships visually and help with covariate selection. The weights can then be used in a regression model to obtain a causal estimate. R was used to create the DAG, IPW, and regression models. This was done with three packages, "dagitty," "twangContinous," and "survey." Three years' worth of NORC's AmeriSpeak survey completion data was used to allocate an appropriate incentive amount to meet a client's completion rate requirement for a survey. AmeriSpeak is a nationally representative probability-based panel of survey respondents from households across the U.S. The survey completions records come from an administrative database, where the intent is for record-keeping, not for conducting research. A DAG was created by combining the survey designs and knowledge of how the data were stored. The DAG helped inform the choice of covariates in creating the IPWs. Finally, the weights were used in a regression model to create reliable incentive effects on completion rates. The estimate is compared to a conventional approach model.

Causal Inference

Inverse Probability Weights

Covariate Selection

Treatment Effects

Survey Incentives

Completion Rates 

Beginner

Practice and Applications

Symposium on Data Science and Statistics (SDSS) 2023