Model-free selective inference with conformal prediction

Music City Center 

Artificial Intelligence (AI) has revolutionized decision-making and scientific discovery in fields like drug discovery, marketing, and healthcare. To ensure the reliability of these models, uncertainty quantification methods such as conformal prediction aim to build prediction sets covering unknown labels of new data. These methods provide on-average (marginal) guarantees which, despite being useful, can be insufficient in decision-making processes that usually come with a selective nature. For instance, early stages of drug discovery aim to identify a subset of promising drug candidates rather than assessing an "average" instance.

We introduce Conformal Selection, a novel framework that offers selective inference capabilities to conformal prediction to address these challenges. We focus on applications where predictions from black-box models are used to shortlist unlabeled test samples whose unobserved outcomes satisfy a desired property, such as identifying drug candidates with high binding affinity. Existing methods based on conformal prediction sets can neglect the selection bias, leading to high fraction of false leads in shortlisted candidates.

Leveraging a set of labeled data that are exchangeable with the unlabeled test points, our method constructs conformal p-values that quantify the confidence in unobserved large outcomes. It then uses the Benjamini–Hochberg (BH) procedure to select the promising candidates whose p-values fall below a data-dependent threshold. We show that this procedure provides finite-sample, distribution-free FDR control.

In addition, I will talk about extensions of the conformal selection method to address the challenges of distribution shift and model selection for optimal performance. One extension, called weighted conformal selection, achieves FDR control when there is a covariate shift between the calibration and test data. Another extension, called optimized conformal selection, maintains FDR control even though the data are reused for selecting a data-dependent, best-performing conformity score. I'll also demonstrate practical applications of the framework in drug discovery and alignment of large language models.

This is based on my PhD work with Emmanuel Candès and new works shortly after.