Evaluating ML Approaches for Assessing Heterogeneity of Treatment Effect in Clinical Trials

2690 

Contributed Abstract 

Poster 

Lisa Levoir (1), Andrew Spieker (2), Bryan Blette (1)

(1) N/A, N/A, (2) Vanderbilt University Medical Center, N/A

Andrew Spieker  
Vanderbilt University Medical Center

Bryan Blette  
N/A

Lisa Levoir  
N/A

Lisa Levoir  
N/A

Inferring heterogeneity of treatment effect is a popular secondary aim of clinical trials. While there are several methods available to estimate conditional average treatment effects (CATEs) in clinical trials, they are often applied in settings with lower sample sizes than were included in corresponding seminal methodological work, making the validity of inference in these settings unclear. To provide practical guidance, we conducted a simulation study to evaluate the performance of different estimators for the CATE, including ordinary least squares (OLS) and causal forests, in a variety of settings. We evaluated 95% confidence interval coverage, bias, and variance under linear and non-linear data generating mechanisms (DGM) in the presence of 0-40 nuisance covariates and 0-16 effect modifying covariates. We found that while tree-based ensembles like causal forests can be quite flexible to linear or nonlinear settings, they can have meaningfully impaired coverage in many settings at sample sizes which constitute most trial applications. As expected, OLS has superior performance under linear DGMs but poor performance under nonlinear DGMs. We conclude with recommendations.

heterogeneous treatment effects|causal forests|machine learning|simulation|causal inference|

Biometrics Section

Personalized/Precision Medicine