General Frameworks for Conditional Two-Sample Testing

Music City Center 

We address the problem of conditional two-sample testing, which assesses whether two populations share the same distribution after accounting for confounding variables. This problem is critical in applications such as domain adaptation and algorithmic fairness, where valid group comparisons must account for such factors. We establish a theoretical hardness result, showing that significant power against any single alternative is unattainable without appropriate assumptions. To address this, we propose two general frameworks: the first transforms any conditional independence test into a conditional two-sample test while preserving its asymptotic properties, and the second leverages estimated density ratios to compare marginal distributions using existing methods for marginal two-sample testing. We demonstrate these frameworks concretely using classification and kernel-based methods, supported by simulation studies to illustrate their efficacy in finite-sample scenarios.

Conditional independence testing

Covariate shift

Density ratio estimation

Algorithmic fairness

Domain adaptation 

Section on Nonparametric Statistics