Adaptive sample splitting for randomization tests

Music City Center 

Randomization tests are widely used to generate finite-sample valid p-values for causal inference on experimental data. However, when applied to subgroup analysis, these tests may lack power due to small subgroup sizes. Incorporating a shared estimator of the conditional average treatment effect (CATE) can substantially improve power across subgroups but requires splitting the treatment assignments between testing and estimation to preserve validity. Motivated by this insight, we introduce AdaSplit, an adaptive sample-splitting procedure that allocates units based on a certainty score for each unit's treatment assignment, computed from its covariates and outcome. The design of AdaSplit is guided by our theoretical analysis, which shows that assignments with high certainty are more effective in increasing test power, while uncertain ones are more valuable for improving CATE estimation when the reserved assignments for randomization tests are imputed from covariates and outcomes. To evaluate the performance of AdaSplit, we conduct simulation studies demonstrating that it yields more powerful randomization tests than baselines that omit CATE estimation or rely on random sample-splitting. Finally, we apply AdaSplit to a blood pressure intervention trial, identifying patient subgroups with significant treatment effects.