Multi-Source Conformal Inference Under Distribution Shift

2419 

Contributed Abstract 

Paper 

Yi Liu (1), Alexander Levis (2), Sharon-Lise Normand (3), Larry Han (4)

(1) North Carolina State University, N/A, (2) Carnegie Mellon University, N/A, (3) Harvard Medical School, N/A, (4) Northeastern University, N/A

Alexander Levis  
Carnegie Mellon University

Sharon-Lise Normand  
Harvard Medical School

Larry Han  
Northeastern University

Yi Liu  
North Carolina State University

Larry Han  
Northeastern University

Recent years have seen a growing utilization of machine learning models to inform high-stakes decision-making. However, distribution shifts and privacy concerns make it challenging to achieve valid inferences in multi-source environments. We generate distribution-free prediction intervals for a target population, leveraging multiple potentially biased data sources. We derive the efficient influence functions for the quantiles of unobserved outcomes and show that one can incorporate machine learning prediction algorithms in the estimation of nuisance functions while still achieving parametric rates of convergence. Moreover, when conditional outcome invariance is violated, we propose a data-adaptive strategy to weight data sources to balance efficiency gain and bias reduction. We highlight the robustness and efficiency of our proposals for a variety of conformal scores and data-generating mechanisms via extensive synthetic experiments and real data analyses.

Conformal prediction|Distribution shift|Federated learning|Missing data|Machine learning|Data integration

IMS

Statistical Methodology