Quantiles, Prediction Intervals, and Probabilistic Regression

Two institutions that could have asked for an expected value asked for a quantile instead.

Flood maps are not drawn around an average. US regulation defines the base flood as the flood with a one percent chance of being equalled or exceeded in any given year. The Federal Emergency Management Agency does not map the average annual flood at all.

Bank supervisors do not compare a trading desk's average loss with its average forecast either. Since January 1996 the Basel Committee on Banking Supervision has had them count instead. Across 250 trading days, how often did the day's loss exceed the bank's own 99th-percentile VaR?

Both are decisions about a tail. Neither can be read off a central value. Two models with the same mean error can produce very different uncertainty ranges. Only one of them may put probability mass where the decision is actually made.

Probabilistic regression evaluates distributions, quantiles, or intervals rather than one central value.

A quantile target penalizes over- and under-prediction differently. One competition made that asymmetry the entire basis of its ranking.

The Global Energy Forecasting Competition 2014 ran four tracks: load, price, wind and solar. It drew 581 contestants from 61 countries. All four tracks were scored the same way. Each contestant submitted the 1st, 2nd, …, 99th percentiles for every period of the forecast horizon. The pinball loss was then averaged over those 99 target quantiles.

The reason was plain. The organisers wrote in 2016: “Considering ease of implementation and communication, we decided to use the pinball loss function, which is closely related to the CRPS anyway.”

Ease of implementation. That is what put one asymmetric loss in charge of a whole competition.

The steps below are what that one score did, ninety-nine times over, to every submission it ranked.

The word “interval” is dangerously overloaded. The three columns below turn up inside the same documents. A bank's 99th-percentile VaR figure is a conditional quantile. The US COVID-19 Forecast Hub required a 95% prediction interval from every team, judged by how often the observed weekly count fell inside it. The standard error printed beside a fitted coefficient is neither of those. Nothing in the output marks which is which.

Coverage is a count, and two institutions publish theirs.

Banks have been counting since 1996. The Basel Committee's backtesting framework is dated 4 January 1996. It compares 250 days of trading outcomes against the bank's 99th-percentile VaR, then sorts the exception count into three zones: green (0–4), yellow (5–9) and red (ten or more). The boundaries are probability statements rather than opinions: “For 250 observations, it can be seen that five or fewer exceptions will be obtained 95.88% of the time when the true level of coverage is 99%.”

The US implementation writes the same counts into law. Under 12 CFR § 3.204 the capital multiplication factor rises from 3.00 at four or fewer exceptions to 3.40 at five and 4.00 at ten or more. Miscalibration has a price list.

Epidemic forecasting ran the same test on many models at once, and printed the result as a table. Cramer and colleagues reported in PNAS in 2022: “Across 1- through 4-wk-ahead horizons, 79 wk, and 50 states, only the ensemble model achieved near-nominal coverage rates for both the 50% and 95% PIs.” Observed coverage was typically below nominal. The rows below come from their Table 1.

Log score, continuous ranked probability score, and related proper scores reward distributions that place probability mass honestly. They can compare more than one interval or quantile. A scalar proper score still needs calibration plots, coverage by slice, and decision analysis. Different scores emphasize different aspects of the distribution.

The goal behind all of them has a standard formulation. Gneiting and colleagues set it out in 2007: evaluate a forecast by “maximizing the sharpness of the predictive distributions subject to calibration”. Calibration is the constraint. Sharpness is the thing being pushed. Calibration is also a joint property, the statistical consistency between forecasts and observations. Sharpness is “a property of the forecasts only”.

Forecasts often arrive as a set of central prediction intervals rather than full densities. A goal of that shape needs a score of the same shape. Bracher and colleagues supplied one in 2021, and their abstract says what it does: “Specifically, we discuss the computation and interpretation of the weighted interval score, which is a proper score that approximates the continuous ranked probability score. It can be interpreted as a generalization of the absolute error to probabilistic forecasts and allows for a decomposition into a measure of sharpness and penalties for over- and underprediction.”

That decomposition is why the score is reported next to coverage rather than instead of it. The sharpness term and the two penalty terms say which way a model is failing. The coverage column says how far.

It became the headline score of the COVID-19 forecast hubs. A 2023 study applied it across 32 countries and restated the property being relied on: the score “is a strictly proper scoring rule, that is, it is optimal for predictions that come from the data-generating model”. Coverage was still assessed separately, at the k = 0.5 and k = 0.95 levels.

Treat central tendency, tails, and decisions as separate views. None of the five steps below is an aspiration. Each one is something a named programme already performs on a schedule.

Step 1 is a submission check that runs automatically. The European COVID-19 Forecast Hub enforced a monotonic increase of predictions with each increasing quantile before a forecast was accepted.

Step 2 is the Basel exception count, codified in the US as 12 CFR § 3.204(b). A bank “must compare each of its most recent 250 business days’ trading losses … with the corresponding daily VaR-based measures calibrated to a one-day holding period and at a one-tail, 99.0 percent confidence level”. The multiplication factor then steps 3.00, 3.40, 3.50, 3.65, 3.75, 3.85 and 4.00 as exceptions run from four or fewer up to ten or more.

Step 3 is the sharpness term inside the weighted interval score.

Step 4 is the per-model coverage column in Table 1 of the COVID-19 hub evaluation. A relative WIS of 0.75 and 52% coverage at the nominal 95% level sit in the same row.

Step 5 is the flood map. The National Flood Insurance Program acts on the 1-percent annual exceedance probability level, not on the expected one.