Evaluating Uncertainty Calibration in Probabilistic Time Series Foundation Models

AI summary

Overview

Research area: Uncertainty quantification and evaluation for time series foundation models (TSFMs), with a focus on probabilistic calibration in multi-horizon forecasting.

Technical level: Intermediate. The paper is written for readers who know basic forecasting and probability concepts (quantiles, prediction intervals, coverage), but it explains its formal definitions as it goes.

Scope: A methodological and empirical study that introduces a taxonomy of how TSFMs represent uncertainty, a multi-level calibration evaluation framework, theoretical limits of post-hoc recalibration, and a benchmark of 14 zero-shot TSFMs on 55 GIFT-Eval dataset configurations.

What This Paper Is About

Time series foundation models increasingly output probabilistic forecasts, but it is unclear whether their stated confidence levels can be trusted — for example, whether a 90% prediction interval really contains the outcome about 90% of the time. Standard evaluations collapse all forecast horizons and all predictions into one pooled number, which can hide systematic errors at particular horizons or at particular predicted values. The paper builds a finer-grained framework for measuring this kind of hidden miscalibration, then tests which post-hoc recalibration methods can actually fix which problems.

Key Contributions

1. A unifying taxonomy of uncertainty mechanisms in TSFMs. The authors organize probabilistic TSFMs into four categories based on the intermediate predictive object from which horizon-wise predictive CDFs are derived: parametric density models, discretized distribution models, direct quantile forecasters, and implicit generative process models. Each category is linked to its training loss (negative log-likelihood, cross-entropy, pinball / weighted quantile loss, flow matching / diffusion denoising) and to representative models.

2. A multi-horizon calibration evaluation framework. The framework distinguishes pooled calibration (PCE_pool), horizon-wise calibration (PCE_avg, PCE_max), and the stronger quantile-conditioned calibration (QCE_avg, QCE_max), which asks whether coverage holds conditional on the model's own predicted quantile value.

3. Theoretical results on the metric hierarchy and on recalibration limits. Proposition 4.2 establishes PCE_pool ≤ PCE_avg ≤ QCE_avg ≤ QCE_max, motivating two diagnostic gap measures — a horizon gap and a conditioning gap. Theorem 4.4 shows that a single global relabeling map can in principle drive pooled error to zero, yet cannot push horizon-wise error below half the original horizon gap, and can even worsen the quantile-conditioned error.

4. A broad empirical study of calibration and recalibration. 14 zero-shot TSFMs across all four taxonomy categories are evaluated on 55 GIFT-Eval dataset configurations, with a separate recalibration experiment (isotonic recalibration, Platt scaling, histogram binning, KDE-based calibration) fitted on the earliest 20% of forecast windows and evaluated on later windows.

Main Findings

• Pooled metrics systematically understate miscalibration: PCE_avg exceeded PCE_pool in 98% of model–dataset pairs, by 40% on average. All points in the PCE_pool-versus-QCE_avg scatter lie above the diagonal, consistent with the stated hierarchy.

• Gaps are statistically significant for most cases: Using the 49 configurations with at least two series, the horizon gap was significant for 86% of model–dataset pairs and the conditioning gap for 69% (level 0.05).

• Miscalibration differs across model families and datasets: Parametric and direct quantile models tended to show the strongest calibration profile, though differences remained dataset-dependent. Ranges in Table 2 include Moirai-Base (PCE_pool 0.0536, PCE_avg 0.0829, QCE_avg 0.1215), Chronos-T5-Base (0.1024, 0.1240, 0.1491) and Sundial-Base (0.1467, 0.1678, 0.1958).

• Some miscalibration is mostly marginal; some is structural: Chronos-T5 and Sundial combined large pooled errors with small corrected conditioning gaps, suggesting mostly marginal miscalibration — Sundial's nominal 80% interval covered only 40% of outcomes. In contrast, Chronos-T5-Small on LOOP_SEATTLE/5T had 29% of speeds exceed the predicted 90% quantile instead of 10%, and Moirai-Small on kdd_cup_2018/H had 27% of outcomes fall below the predicted 10% quantile — errors of opposite sign that cancel in marginal coverage.

• Recalibration helps average calibration most: Isotonic recalibration reduced PCE_pool by 37%, PCE_avg by 19% and QCE_avg by 15%, while the horizon gap increased by 24% and the conditioning gap was essentially unchanged. PCE_avg improved for 62% of model–dataset pairs, with the largest gains where original miscalibration was most severe.

• Benefits are concentrated in models with marginal miscalibration: Chronos-T5, Sundial and TimesFM lost 51–62% of their pooled error, versus 9–24% for the Moirai and Chronos-Bolt models. In Table 3, Moirai-Base under isotonic recalibration improved PCE_pool by 13.36%, PCE_avg by 4.77% and QCE_avg by 5.79%, while Chronos-T5-Base improved PCE_pool by 62.23%.

• Value-dependent error survives global recalibration: For the two structured cases above (conditioning gaps of 0.039 and 0.044, both significant), recalibration moved the gaps only from 0.041 to 0.038 and from 0.040 to 0.038 on test windows, because one map applies to all forecasts.

• Interval-level fixes can be large: For Chronos-2 on LOOP_SEATTLE/D, 90% intervals covered only 77% of later outcomes; isotonic recalibration raised this to 91%.

Methodology in Plain English

The authors first sort existing TSFMs by how they produce uncertainty — a distribution family with parameters, probabilities over value bins, a fixed set of predicted quantiles, or sampled forecast trajectories. Since all four eventually yield a predicted distribution for each forecast step, calibration can be measured on a shared quantile grid of {0.1, 0.2, ..., 0.9}, extracting quantiles from each model's output or samples where necessary.

They then measure calibration at three increasing levels of strictness. The loosest averages over all horizons and instances at once. The middle level measures error separately at each forecast step and reports the mean and worst-case. The strictest asks whether coverage still holds when forecasts are grouped by the value the model itself predicted, estimated nonparametrically via isotonic regression, with resampling-based significance tests for the two gap diagnostics.

Finally, they simulate deployment by fitting each recalibration method on the earliest 20% of forecast windows per series (or 20% of series for single-window datasets) and scoring it on the remaining later windows, comparing relative changes per metric.

Why This Matters

For research, the paper shows that the calibration numbers commonly reported for TSFMs are the weakest in a hierarchy, and it supplies a formal reason why global recalibration cannot fix horizon-specific or value-dependent errors. It gives the field a shared vocabulary (horizon gap, conditioning gap) and a benchmark protocol rather than a single aggregate score.

Real-world applications named in the paper:

• Healthcare, where forecasts inform high-stakes decisions and adverse-outcome likelihoods matter.
• Energy systems, where planning depends on the probability of extreme demand or supply conditions.
• Finance, where risk assessment depends on the stated tails of a forecast distribution.
• General short-term forecasting on heterogeneous domains and sampling frequencies, as represented by the GIFT-Eval configurations used.

Industry relevance: TSFMs are typically large, pretrained, and used zero-shot, so retraining is expensive. The paper's finding that lightweight post-hoc relabeling is data-efficient — fitted on just 20% of windows — but corrects only average misalignment is directly actionable for practitioners deciding whether a recalibration step is enough for a deployment.

Future Directions

• Recalibration that conditions on the forecast itself. The authors note that removing value-dependent error would require recalibration conditioning on the predicted value rather than a single global map, which their results motivate but do not implement.

• Horizon-specific correction. Since no global relabeling can reduce horizon-wise error below half the original horizon gap, methods that treat forecast steps separately, or that model dependence across the horizon, remain an open problem.

• Extending beyond marginal horizon-wise CDFs. The analysis deliberately restricts attention to per-step marginal distributions; calibration of the joint distribution across horizons is not addressed.

• Broadening the empirical base. Results are reported for 55 dataset configurations (42 for recalibration, 49 for significance testing) on the short-term GIFT-Eval task; whether the observed patterns hold for other benchmarks, longer horizons, or fine-tuned rather than zero-shot models is not established.

Target Audience

Researchers and practitioners working on probabilistic forecasting and foundation models who need to evaluate or report uncertainty quality; machine learning engineers deploying TSFMs in risk-sensitive settings; and statisticians interested in calibration theory for multi-horizon and conditional notions of reliability. Readers should be comfortable with quantiles and prediction intervals, though the paper defines its formal concepts as it introduces them.

Authors’ abstract

Time series foundation models (TSFMs) are increasingly used for probabilistic forecasting across diverse domains, yet the reliability of their uncertainty estimates remains poorly understood. In this work, we study uncertainty calibration in probabilistic TSFMs from a broader methodological and empirical perspective. We introduce a taxonomy of how TSFMs represent predictive uncertainty and develop a fine-grained evaluation framework for calibration in multi-horizon forecasting that distinguishes pooled, horizon-wise, and stronger conditional notions of calibration. Using this framework, we conduct a comprehensive empirical evaluation of popular TSFMs across diverse datasets and show that current models exhibit distinct and systematic patterns of miscalibration that standard metrics often obscure. We further analyze common post-hoc recalibration methods and characterize which forms of calibration they can improve and which tend to persist. Our results provide practical guidance for evaluating and improving uncertainty calibration in TSFMs and highlight important limitations of current practice.