Calibrated multi-level quantile forecasting

Music City Center 

In order for probabilistic forecasts to be useful to decision makers, the forecasts should be calibrated – given a sequence of 90% quantile forecasts, we want the true value to be less than the forecast 90% of the time. Existing online calibration procedures, such as the quantile tracking algorithm from online conformal prediction (Angelopoulos et al., 2023), are able to effectively calibrate a single quantile but, when applied to multiple quantiles, can produce invalid probability distributions due to crossings – e.g., the calibrated 50% quantile forecast is above the calibrated 75% quantile forecast. In this work, we consider the problem of online calibration with order constraints. We propose intuitive ways of combining the quantile tracking algorithm with an order-enforcing method (such as sorting or isotonic regression) that produce a sequence of forecasts with no crossings but is also guaranteed to achieve the correct long-run coverage under mild assumptions. We demonstrate our methods on COVID-19 forecasting data.

forecasting

calibration

conformal prediction

online learning 

IMS