Hypothesis testing approaches for differences in paired AUCs under nested cross-validation

Thomas M. Menino Convention & Exhibition Center 

Sample size can often be limited in various AI/ML modeling contexts where the objective is to develop a binary classification model and where discrimination performance is summarized using area under the ROC curve (AUC). Under these conditions, nested cross-validation (NCV) may be used to examine performance summarized across outer folds versus a standard single train-test split. When there are multiple competing models, there may be additional interest in comparing the respective performances. A naive testing strategy would be to apply standard paired DeLong testing using stacked results across the NCV outer folds. However, this strategy results in overly conservative variance estimates and inflates Type I error; in contrast, bootstrapping the full training process for valid variance estimates may not be computationally feasible for more demanding algorithms. Here, we conduct simulations to examine multiple competing strategies for paired AUC comparisons under NCV, including a novel approach based on influence functions. By considering different outer fold counts, ML algorithms, and target AUC values, we illustrate Type I error and power under comprehensive testing conditions.

Machine learning

AUC

Cross-validation

Hypothesis testing 

Section on Statistical Learning and Data Science