Benchmarking Conformal Inference Under Covariate Shift

Thomas M. Menino Convention & Exhibition Center 

Conformal prediction has emerged as a robust framework for providing prediction uncertainty quantification with applications in large language models and genomics. However, these guarantees often rely on the exchangeability assumption, which is frequently violated in real-world scenarios due to covariate shift. While several adaptations maintain validity under distribution shift, the literature lacks a comprehensive, unified evaluation of their performance across diverse data regimes.

This paper presents a rigorous benchmark of state-of-the-art methods for conformal inference under covariate shift. We evaluate Weighted Conformal Prediction (Tibshirani et al.), Conformal Prediction Under Covariate Shift (Park et al.), Robust Conformal Prediction (Chernozhukov et al.), Entropy Balancing, Covariate Shift through Optimal Transport (Giguere et al.), Nex-CP (Barber et al.), and Conformal Prediction with Conditional Density Estimation (Borgwardt et al.).

Our study assesses these methods across high-dimensional synthetic and real-world datasets, focusing on coverage adherence, interval efficiency, and scalability. By identifying the strengths and failure modes of each approach, this

Conformal Inference

Covariate Shift

Conditional Coverage 

Section on Statistical Learning and Data Science