Advancing Pharmaceutical Research: Estimating Heterogenous Treatment Effects through Virtual Twins

Secondary analyses of randomized clinical trials often seek to identify subgroups with differential treatment effects. These discoveries can help guide individual treatment decisions based on patient characteristics and identify populations for which additional treatments are needed. Traditional analyses require researchers to pre-specify potential subgroups to reduce the risk of reporting spurious results. There is a need for methods that can detect such subgroups without a priori specification while allowing researchers to control the probability of falsely detecting heterogeneous subgroups when treatment effects are uniform across the study population. We propose a permutation procedure for tuning parameter selection that allows for type I error control when testing for heterogeneous treatment effects framed within the Virtual Twins procedure for subgroup identification. We verify that the type I error rate can be controlled at the nominal rate and investigate the power for detecting heterogeneous effects when present through extensive simulation studies. We apply our method to a secondary analysis of data from a randomized trial of very low nicotine content cigarettes. In the absence of type I error control, the observed type I error rate for Virtual Twins was between 99% and 100%. In contrast, models tuned via the proposed permutation were able to control the type I error rate and detect heterogeneous effects when present. An application of our approach to a recently completed trial of very low nicotine content cigarettes identified several variables with potentially heterogeneous treatment effects. The proposed permutation procedure allows researchers to engage in secondary analyses of clinical trials for treatment effect heterogeneity while maintaining the type I error rate without pre-specifying subgroups. 

Inferring causal effects from observational studies is a key focus in various scientific fields, including social science, healthcare, and medicine. While statistical methodologies for estimating the population average causal effect are well-established, techniques for identifying and estimating subpopulation causal effects are comparatively less developed. A significant challenge is that subgroup structures are often unknown, requiring adaptations to methods designed for population-level inference.
We propose a tree-based method, built on a matched design, to identify subgroups with differential treatment effects. To address observed confounding, we first create propensity-score-matched pairs. Next, we apply classification and regression trees (CART) to the differences in outcomes within matched pairs, uncovering subgroup structures with distinct causal effects. This nonparametric approach is robust against model misspecification—an essential feature given the difficulty of specifying parametric outcome models in the presence of complex subgroup effects.
We outline the assumptions under which the proposed matching estimator remains unbiased and provide algorithms for identifying subgroup structures. Simulations demonstrate that our method outperforms competing tree-based approaches—including causal trees, causal inference trees, and the virtual twins approach—in accurately identifying the true subgroup structure. Finally, we apply our method to evaluate the potential subgroup effect of Tobramycin timing on chronic infection outcomes among pediatric Cystic Fibrosis patients.
 

Successful clinical development depends on a thorough understanding of relevant internal and external clinical trial data, along with published literature. By leveraging this information, researchers can gain deeper insights into heterogeneous patient populations and the covariates influencing treatment efficacy, ultimately enhancing the likelihood of program success and facilitating efficient decision-making. Popular methods such as virtual twins along with machine learning (ML), Bayesian methods, causal inference can be utilized to assess treatment effects. The integration of ML techniques with Bayesian methods allows for accurate predictions of treatment efficacy while also providing uncertainty estimates for model outputs, which is essential for informed decision-making and risk assessment. Additionally, some limitations of existing algorithms, such as model misspecification, can be mitigated by incorporating approaches from the increasingly prominent field of causal inference.
In this talk, we will explore the application of virtual twins incorporating Bayesian machine learning techniques to both simulated and real clinical trial data, focusing on their performance from a causal inference perspective. Specifically, we will investigate three machine learning algorithms—Random Forest, elastic net, and artificial neural networks— together with virtual twins to identify heterogeneous subpopulations. Our findings will demonstrate that common challenges faced by traditional ML algorithms, such as low prediction accuracy, overfitting, and insufficient uncertainty estimates, can be effectively addressed through the integration of Bayesian methodologies to improve traditional virtual twin framework. Additionally, we will provide a statistically rigorous uncertainty quantification through conformal prediction interval. 

When heterogeneous treatment is present, averaged treatment effect (ATE) is insufficient and even misleading for informing patient-centered treatment effect (PCTE). The ATE over nearest neighbors (NN) could offer a relative homogeneous sample that resembles the patient at the "center" of treatment decision. The k-nearest neighbor algorithm (kNN; Fix and Hodges, 1951) is a popular nonparametric method for supervised learning as it provides asymptotic uniform probability density over a given neighborhood (Silverman & Johns, 1989). The kNN has been used in matching based causal inference (Stuart, 2010). Applying kNN for identifying "alike" patients, Zhou and Kosorok (2017) shown the causal k-nearest neighbor (kNN) regime is universally consistent in the sense that it will learn the optimal treatment regime as the sample size increase. This study evaluates the leave-one-out (LOO) performances of some popular NN causal methods via simulation – kNN-ATE, kNN-IPTW, honest causal forest (HCF) and Bayesian adaptive regression tree (BART). The LOO performances are summarized by the quartiles of absolute error (Q1, Q3 and Median AE or MAE), root mean square predicted error (rMSPE) and averaged coverage rate (CR) over all LOO estimates compared against the oracle distributions for each LOO unit. Case study applied the methods to electronic health records to evaluate patient-centered treatment effect for children newly diagnosed with Juvenile Idiopathic Arthritis (JIA), a highly heterogeneous disease condition without known etiology. The study suggests the performances of all methods vary by the data density within NN, as well as its functional complexity of PCTE. Methods to evaluate and validate performances of PCTE method should look carefully into pairwise performances of LOO estimates. The NN causal is a useful approach to learn from real-world-data (RWD) about PCTE.  

Individuals can vary drastically in their response to the same treatment, and this heterogeneity has driven the push for more personalized medicine. Accurate and interpretable methods to identify subgroups that respond to the treatment differently from the population average are necessary to achieving this goal. The Virtual Twins (VT) method is a highly cited and implemented method for subgroup identification because of its intuitive framework. However, since its initial publication, many researchers still rely heavily on the authors' initial modeling suggestions without examining newer and more powerful alternatives. This leaves much of the potential of the method untapped. We comprehensively evaluate the performance of VT with different combinations of methods in each of its component steps, under a collection of linear and nonlinear problem settings. Our simulations show that the method choice for Step 1 of VT, in which dense models with high predictive performance are fit for the potential outcomes, is highly influential in the overall accuracy of the method, and Superlearner is a promising choice. We illustrate our findings by using VT to identify subgroups with heterogeneous treatment effects in a randomized, double-blind trial of very low nicotine content cigarettes.