Characterizing Retrospective Constraints of Survey Designs for Estimating Population-level Causal Effects

Thomas M. Menino Convention & Exhibition Center 

Evaluating causal effects in a general population is a critical step for decision makers. Large general use population surveys provide a unique data source for extracting causal evidence. A key limitation is that most surveys are designed primarily for accurate population description rather than causal effect estimation. This paper seeks to characterize some retrospective constraints of survey data for post-hoc causal effect estimation. First, assuming multi-phase selection from a finite population of potential outcomes, we give examples of structural scenarios where construction of the survey weights may be informative for exposure status, leading to biased effect estimates. We then show that a post-hoc weight class adjustment may be applied to reduce bias in this setting. Further, sensitivity analysis relaxing the ignorability assumption is important for observational studies. However, sensitivity models for weighting estimators based on the percentile bootstrap may be anticonservative if the percentile bootstrap is not adapted to the complex survey design. We show that survey bootstrap techniques can properly account for this sampling variation, leading to valid confidence intervals. Simulation studies demonstrate the superior empirical performance of our approach over conventional methods. Finally, we illustrate these concepts using the National Youth and Tobacco Survey to estimate the causal effect of e-cigarette use on the future intention to smoke conventional cigarettes among middle and high school students.

Causal inference

complex survey design

sensitivity analysis

generalizability