Multi-faceted Approaches to Sensitivity Analysis for Observational Studies

Sensitivity analyses can inform evidence-based education policy by quantifying the hypothetical conditions necessary to change an inference. Perhaps the most prevalent index used for sensitivity analyses is Oster's (2019) Coefficient of Proportionality (COP). Oster's COP leverages changes in estimated effects and R2 when observed covariates are added to a model to quantify how strong selection on unobserved covariates would have to be relative to on observed covariates to nullify an estimated effect. In this paper, we reconceptualize the COP as a function of unobserved covariates' correlations with the focal predictor (e.g., treatment) and with the outcome. Our correlation-based approach addresses recent critiques of Oster's COP while preserving the comparison of selection on unobserved covariates to selection on observed covariates. As importantly, our expressions do not depend on an analyst's subjective choice of covariates to include in a baseline model, are exact even in finite samples, and can be directly calculated from conventionally reported quantities (e.g., estimated effect, standard error) through the Konfound packages in R or Stata. Thus, for most published studies in the social sciences our COP index can be easily applied and intuitively interpreted. 

Mediation analysis is an important part of social science research, which helps examine the extent to which the effect of a treatment on an outcome operates through mediators of interest (VanderWeele, 2015). A key assumption for identifying causal mediation effects, such as the interventional indirect effects (Vansteelandt & Daniel, 2017), is the absence of unmeasured pretreatment confounders of the mediator-outcome relation. This assumption is difficult to satisfy even in randomized controlled trials.

Various sensitivity analysis methods have been developed to assess the impact of unmeasured confounders on mediation results (Ding & Vanderweele, 2016; Hong et al., 2018; Imai et al., 2010; Park & Esterling, 2021; Rubinstein et al., 2023; Tchetgen Tchetgen & Shpitser, 2012; VanderWeele, 2010; Zhang & Ding, 2023). A majority of existing sensitivity analysis methods focus on a single mediator and non-clustered data. Mediation analyses in social science, however, often involve multiple mediators and require accounting for clustered data (e.g., students clustered in schools), yet sensitivity analysis methods for this setting remain scarce (Qin et al., 2021). Furthermore, the use of parametric models for causal effect estimation has become increasingly concerning (Ogburn & Shpitser, 2021), and recent years have seen growing use of machine learning methods in estimation of causal effects (Kennedy, 2024; van der Laan & Rose, 2018). Despite this, tools for assessing sensitivity to unmeasured confounders in mediation analyses using machine learning based estimators are lacking.

This study presents a sensitivity analysis method for causal mediation analysis with multiple mediators and clustered data, incorporating nonparametric estimation based on doubly robust machine learning methods (Benkeser & Ran, 2021; Chernozhukov et al., 2018, 2024; Liu et al., 2024; Rubinstein et al., 2023). Under the assumption of no interference, we examine the influence of unmeasured pretreatment confounders in the mediator-outcome relationship on inferences of interventional indirect effects. Using data from the Educational Longitudinal Study (ELS; Ingels et al., 2007), we illustrate our sensitivity analysis approach in assessing the interventional indirect effects of extracurricular activity participation (treatment) on student academic achievement (outcome) via two mediators related to student educational aspirations and beliefs. We hope this study adds to researchers' toolbox for quantifying the sensitivity of causal mediation analysis results to unmeasured confounders.



References

Benkeser, D., & Ran, J. (2021). Nonparametric inference for interventional effects with multiple mediators. Journal of Causal Inference, 9(1), 172–189. https://doi.org/10.1515/jci-2020-0018
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097
Chernozhukov, V., Cinelli, C., Newey, W., Sharma, A., & Syrgkanis, V. (2024). Long story short: Omitted variable bias in causal machine learning (arXiv:2112.13398). arXiv. https://doi.org/10.48550/arXiv.2112.13398
Ding, P., & Vanderweele, T. J. (2016). Sharp sensitivity bounds for mediation under unmeasured mediator-outcome confounding. Biometrika, 103(2), 483–490. https://doi.org/10.1093/biomet/asw012
Hong, G., Qin, X., & Yang, F. (2018). Weighting-based sensitivity analysis in causal mediation studies. Journal of Educational and Behavioral Statistics, 43(1), 32–56. https://doi.org/10.3102/1076998617749561
Imai, K., Keele, L., & Yamamoto, T. (2010). Identification, inference and sensitivity analysis for causal mediation effects. Statistical Science, 25(1), 51–71. https://doi.org/10.1214/10-STS321
Ingels, S. J., Pratt, D. J., Wilson, D., Burns, L. J., Currivan, D., Rogers, J. E., & Hubbard-Bednasz, S. (2007). Education Longitudinal Study of 2002 (ELS: 2002): Base-Year to Second Follow-Up Data File Documentation. NCES 2008-347. National Center for Education Statistics.
Kennedy, E. H. (2024). Semiparametric doubly robust targeted double machine learning: A review. In Handbook of Statistical Methods for Precision Medicine. Chapman and Hall/CRC.
Liu, R., Williams, N. T., Rudolph, K. E., & Díaz, I. (2024). General targeted machine learning for modern causal mediation analysis (arXiv:2408.14620). arXiv. https://doi.org/10.48550/arXiv.2408.14620
Ogburn, E. L., & Shpitser, I. (2021). Causal Modelling: The Two Cultures. Observational Studies, 7(1), 179–183.
Park, S., & Esterling, K. M. (2021). Sensitivity analysis for pretreatment confounding with multiple mediators. Journal of Educational and Behavioral Statistics, 46(1), 85–108. https://doi.org/10.3102/1076998620934500
Qin, X., Deutsch, J., & Hong, G. (2021). Unpacking complex mediation mechanisms and their heterogeneity between sites in a job corps evaluation. Journal of Policy Analysis and Management, 40(1), 158–190. https://doi.org/10.1002/pam.22268
Rubinstein, M., Branson, Z., & Kennedy, E. H. (2023). Heterogeneous interventional effects with multiple mediators: Semiparametric and nonparametric approaches. Journal of Causal Inference, 11(1). https://doi.org/10.1515/jci-2022-0070
Tchetgen Tchetgen, E. J., & Shpitser, I. (2012). Semiparametric theory for causal mediation analysis: Efficiency bounds, multiple robustness and sensitivity analysis. The Annals of Statistics, 40(3), 1816–1845. https://doi.org/10.1214/12-aos990
van der Laan, M. J., & Rose, S. (2018). Targeted Learning in Data Science: Causal Inference for Complex Longitudinal Studies. Springer International Publishing. http://link.springer.com/10.1007/978-3-319-65304-4
VanderWeele, T. J. (2010). Bias formulas for sensitivity analysis for direct and indirect effects. Epidemiology, 21(4), 540–551. https://doi.org/10.1097/EDE.0b013e3181df191c
VanderWeele, T. J. (2015). Explanation in causal inference: Methods for mediation and interaction. Oxford University Press.
Vansteelandt, S., & Daniel, R. M. (2017). Interventional effects for mediation analysis with multiple mediators. Epidemiology, 28(2), 258–265. https://doi.org/10.1097/EDE.0000000000000596
Zhang, M., & Ding, P. (2023). Interpretable sensitivity analysis for the Baron-Kenny approach to mediation with unmeasured confounding (arXiv:2205.08030). arXiv. https://doi.org/10.48550/arXiv.2205.08030


 

Contingency tables are a popular approach to study associations between categorical variables in an observational study. Unfortunately, most association tests for contingency tables assume no unmeasured confounding, an unrealistic assumption, and sensitivity analysis, a popular procedure to assess the bias from unmeasured confounders, often assumes a binary categorical variable. This paper proposes a sensitivity analysis for contingency tables. Specifically, we extend Rosenbaum's sensitivity model to non-binary categorical variables and present an algorithm to compute the exact, worst-case null distribution for a family of association tests. Results accelerating the computation in a sensitivity analysis when the test statistic is (a) permutation-invariant (b) a sum score test with non-binary outcome (c) a sum score test with binary outcome are also presented. We also complement the exact approach with a derivation of the asymptotic null distribution.

We also compare the power to detect an ordinal treatment effect between our approach and two simpler alternatives: (a) a sensitivity analysis that binarizes a multi-level outcome (i.e., binarizing) and (b) a sensitivity analysis using Fisher's exact test, which reduces an I by J contingency table to a 2 by 2 table by combining categories (i.e., collapsing). Our results demonstrate that retaining the original I by J table yields greater statistical power than the two simpler alternatives.

Finally, we extend our findings to stratified I by J tables (i.e., I by I by K tables) and illustrate our approach through a re-analysis of the impact of pre-kindergarten care on math achievement using data from the Early Childhood Longitudinal Study, Kindergarten Class of 1998-1999.
 

A sensitivity analysis assesses the robustness of an observational study's conclusions to unmeasured
confounding of increasing strength. We present a new approach to conducting a sensitivity analysis in matched observational studies within Rosenbaum's sensitivity model. For any candidate test statistic, the approach introduces tilted modifications dependent upon the proposed strength of unmeasured confounding. The framework subsumes both (i) existing approaches to sensitivity analysis for sign-score statistics; and (ii) sensitivity analyses using conditional inverse probability weighting, wherein the researcher weights the observed test statistic based upon the worst-case assignment probabilities for a proposed strength of hidden bias. Unlike the prevailing approach to sensitivity analysis in matched observational studies, there is a closed form expression for the limiting worst-case distribution even when matching with multiple controls. Moreover, the approach admits a closed form solution for its design sensitivity, a measure used to compare competing test statistics and research designs, when matching with multiple controls, whereas the conventional approach only does so for pair matching. The tilted sensitivity analysis improves design sensitivity relative to the conventional approach under a host of generative models. The proposal may also be adaptively combined with the conventional approach to attain a design sensitivity no smaller than the max of the individual design sensitivities. Beyond the aforementioned computational and theoretical benefits, data illustrations indicate that tilting can provide meaningful improvements in reported robustness of matched observational studies to hidden bias. 

Choices about observational study design, notably the choice of estimand, have important implications for whether the final estimate will exhibit robustness to unmeasured confounding. In practice however, the aspects of a study that influence sensitivity to unmeasured confounding are not well understood or accounted for when planning a study. We demonstrate how design sensitivity, a quantity describing the asymptotic power of a sensitivity analysis, can be used to compare multiple candidate estimands in weighted observational studies to improve robustness to unmeasured bias. Specifically, using data from a referendum on the 2016 Colombian peace agreement we explore how altering the definition of treatment and altering the target population of interest impact the expected performance of sensitivity analyses.