016: A Bayesian Approach Towards Balanced Probability Calibration and Boldness

There is a fundamental tension between the calibration and boldness of probability predictions about forthcoming events. Predicted probabilities are considered well calibrated when they are consistent with the relative frequency of the events they aimed to predict. However, well calibrated predictions are not necessarily useful. Predicted probabilities are considered more bold when they are further from the base rate and closer to the extremes of 0 or 1. Predictions that are reasonably bold, while maintaining calibration, are more useful for decision making than those with only one or the other. We develop Bayesian estimation and hypothesis testing-based methodology with a likelihood suited to the probability calibration problem. Our approach effectively identifies and corrects miscalibration. Additionally, it allows users to maximize boldness while maintaining a user specified level of calibration, providing an interpretable tradeoff between the two. While we demonstrate the practical capabilities of this methodology by comparing hockey pundit predictions, this approach is widely applicable across many fields.

Sports Statistics

Calibration

Estimation

Model Selection

Bayesian 

Implementation and Analysis

Conference on Statistical Practice (CSP) 2023