Rethinking Conformal Prediction for Binary Classification

Thomas M. Menino Convention & Exhibition Center 

In binary classification, standard Conformal prediction (CP) often collapses to the uninformative set $\emptyset$ or $\{0, 1\}$. We identify a structural cause: for any nonconformity score monotone in $\hat{p}(y|x)$, a nontrivial fraction of test points can receive two-label prediction sets. We also show that pointwise level shrinkage $\alpha(x)$ under the standard split CP formulation may not achieve conditional validity, yielding a second impossibility result. Motivated by these limits, we propose GLoSaM, a groupwise CP method with data-driven grouping and adaptive calibration that tightens prediction sets while retaining finite-sample distribution-free guarantees. Across synthetic benchmarks and binary LLM and vision classification settings, GLoSaM achieves valid groupwise coverage, substantially higher singleton rates, and robustness to score choice, outperforming e-value and multi-level conformal baselines.

Groupwise conformal prediction

Local adaptivity

Singleton Accuracy

Exchangeability

Conditional validity 

Section on Statistical Learning and Data Science