Regression Recalibration by CRPS Minimization

Oregon Convention Center 

In an era dominated by large-scale machine learning models, poor calibration severely limits the trustworthiness of the results. As we increasingly rely on complex systems, recalibration becomes essential, where the objective is to find a mapping that adjusts the model's original probabilistic prediction to a new, more reliable one. We explore a broad class of recalibration functions based on learning the optimal step function over a proper scoring rule. Using the continuous ranked probability score (CRPS) and applying predicted-mean binning, our approach outperforms the widely-used quantile recalibration method in terms of both calibration and sharpness, while maintaining its simplicity. We apply our method to a case study on the Pinatubo eruption climate dataset using a convolutional neural network model with dropout.