009 - The Siren Song of Propensity Score Methods: An Assessment of PSM in the Context of the E-Value

Data analyses are predicated on assumptions, many of which are not directly testable. A key assumption underlying observational research methods for analysis of real-world data (e.g. health insurance claims or electronic health records) is that treatment assignment is independent of the potential outcomes. In other words, an unbiased estimate of the treatment effect is obtained only when there are no unmeasured confounders that are related to both who receives the treatment and the outcomes of interest. However, it is not possible to directly test for the impact of covariates that are unobserved to the analyst.

Our study uses Monte Carlo simulations to evaluate the impact of unobserved confounders on the treatment effect estimates and to evaluate the performance of the E-Value sensitivity test with the application of regression and propensity score methods under varying levels of unobserved confounding. Specifically, we compare observed and unobserved confounder balance, odds ratios of treatment vs. control, and E-Values sensitivity test statistics from GLM regression models, inverse-probability weighted models, and propensity score matching models, over correlations of increasing strength between observed and unobserved confounders. The E-Value sensitivity test to assess the effect of unmeasured confounding is notable for its ease of implementation and interpretation. The E-value reports the minimum strength of association between an unmeasured confounder and the treatment and outcome that would explain away the estimated treatment effect.

We find that, while propensity score methods balance observed confounders, they may exacerbate imbalances in unobserved confounders resulting in biased treatment effect estimates. Moreover, we find that E-values calculated after applying propensity score methods tend to be larger when unobserved confounders result in more biased treatment effect estimates. This result has important implications for the appropriate application and interpretation of common statistical methods and sensitivity testing. First, we confirm previous findings that the propensity score methods – matching or weighting – may increase the imbalance in unobserved confounders. The magnitude of the effect depends on the strength of correlation between the confounder, treatment, and outcomes. In all cases, this implies that propensity score methods are only appropriate to use when the underlying assumptions can be justified by knowledge of the treatment context and data source. Second, the E-Value may misrepresent the size the unobserved effect needed to change the magnitude of the association between treatment and outcome when propensity score methods are used.

Thus, caution is warranted when interpreting the E-Value. Sensitivity testing is an important element of any analysis but is not a substitute for a well-informed study design.

observational research methods

propensity score methods

simulation

E-value

sensitivity testing 

Mid-Level

Knowledge

International Conference on Health Policy Statistics 2023