Multiclass Recalibration of Probability Predictions via the Linear Log Odds Calibration Function

Thomas M. Menino Convention & Exhibition Center 

Machine-generated probability predictions are essential to modern classification tasks such as image classification. A model is well calibrated when its predicted probabilities correspond to observed event frequencies. Despite the importance of multicategory recalibration, existing approaches are limited by comparing calibration between models rather than assessing the calibration of a single model, requiring under-the-hood access to model access, and producing outputs that are difficult for human analysts to understand. To address these limitations, we propose Multicategory Linear Log Odds (MCLLO) recalibration, which incorporates a likelihood ratio hypothesis test to assess calibration, doesn't require internal model access, and yields interpretable results. We demonstrate the effectiveness of MCLLO through simulations and three case studies involving image classification via convolutional neural network, obesity analysis via random forest, and ecology via regression modeling. We compare MCLLO to four comparator recalibration techniques using both our hypothesis test and the Expected Calibration Error to show that our method works well alone and in concert with other methods.

Multiclass Classification

Calibration

Machine Learning

Confidence Scores

Likelihood Ratio Test 

Uncertainty Quantification in Complex Systems Interest Group