Conformal Mirror Statistics for Model Alignment: Uncertainty Quantification with FDR Control

Music City Center 

Foundation models are increasingly deployed across various domains, offering valuable insights and making decisions. However, ensuring their outputs align with human interpretations is critical before deployment, particularly in high-stakes applications. This highlights the need for a rigorous uncertainty quantification (UQ) method to assess alignment reliability. Most existing methods rely on large labeled training datasets, limiting their applicability in real-world settings where labeled data is scarce or expensive. This paper introduces Conformal Mirror Statistics (CMS), a novel framework for UQ in model alignment. Unlike conventional conformal methods based on p-value calibration, CMS generalizes to broader settings without the restriction of sample size regarding test and calibration sets, while tightly controlling FDR. Empirical evaluations on two large sepsis cohorts from MIMIC-III and IV demonstrate that CMS is able to reliably select candidates with certain outputs while outperforming conventional methods in FDR control.

Conformal Inference

False Discovery Rate Control

Model Alignment

Uncertainty Quantification 

Section on Statistical Learning and Data Science