Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?

AI summary

Overview

Research area: Probabilistic time series forecasting, uncertainty quantification, and model calibration for time series foundation models (TSFMs). Technical level: Intermediate (accessible to readers familiar with basic forecasting concepts such as quantiles and prediction intervals). Scope: A systematic empirical study comparing the calibration of five state-of-the-art time series foundation models against two baselines across six diverse datasets.

What This Paper Is About

Time series foundation models are general-purpose models trained on many different time series that can produce forecasts zero-shot, without retraining per series. These models output full conditional distributions (or sets of quantiles), not just single point forecasts, so a natural question is whether the probabilities they assign actually match the observed data. This paper asks whether TSFMs are well-calibrated, whether they are systematically over- or under-confident, and how design choices such as prediction heads and long-horizon autoregressive forecasting affect calibration.

Key Contributions

1. A first-of-its-kind systematic calibration study of five state-of-the-art time series foundation models (Chronos-Bolt, TimesFM, Moirai 2.0, TiRex, YingLong) and two baselines (ARIMA and N-BEATS) across six univariate datasets spanning different temporal granularities.
2. Introduction of an evaluation based on three calibration-specific metrics — Probabilistic Calibration Error (PCE), Centered Calibration Error (CCE) and Scaled Interval Width (SIW) — alongside Mean Absolute Scaled Error (MASE) for point accuracy, rather than relying on CRPS, WQL or MSIS, which the paper argues conflate calibration with sharpness.
3. An analysis of how different prediction heads (Gaussian, Student's t, mixture, and quantile, all trained on the TSMixup dataset) affect calibration when substituted into each pretrained backbone.
4. An investigation of how long-term autoregressive forecasting — comparing naive, branching and trajectory AR methods across varying forecast horizons — affects calibration.

Main Findings

• TSFMs are better calibrated than baselines. Using PCE with quantiles q ∈ {0.1, 0.2, ..., 0.9}, foundation models generally achieve PCE values close to or below 0.05 (5%) error, whereas ARIMA and N-BEATS have consistently higher PCE. No single TSFM significantly dominates the others on calibration.
• PCE is not the same as WQL. The paper shows that Weighted Quantile Loss can be highly correlated with point accuracy (MASE). On the Glucose dataset, WQL incorrectly identifies ARIMA as the best-calibrated model, illustrating why the authors use calibration-specific metrics.
• TSFMs are not systematically over- or under-confident. CCE results show TSFMs tending to be neither systematically overconfident nor underconfident, in contrast to the overconfidence reported for deep learning models in image and text domains. The authors attribute this to TSFMs being trained directly with a calibration-aware loss (minimizing WQL), whereas image and text models are trained to minimize reconstruction or classification error.
• Baselines are consistently under-confident. N-BEATS and ARIMA show negative CCE, meaning they are systematically under-confident. SIW and CCE tend to be negatively correlated: wider intervals go with lower CCE.
• The Patents dataset is the exception. On Patents, all methods (TSFMs and baselines) have high calibration error, and all models are overconfident. The authors note the Glucose and Patents datasets have significantly higher MASE, possibly because these datasets show little significant linear dependence beyond 1 or 2 lags.
• Calibration degrades with forecast distance. Point accuracy (MASE) and calibration error (PCE) both worsen as models predict further into the future, but TSFMs maintain PCE close to or below 5% even 64 time steps out (e.g., on Reviews, M5 and Crime), unlike baselines which have consistently high PCE at all prediction lengths.
• Control experiments on synthetic data. On pure IID noise (y_t = ε) and a noisy first-order linear process (y_t = α y_{t-1} + (1−α) ε, with ε ~ N(0,1) and α = 0.9), TSFMs do not overfit and perform well on both MASE and PCE. ARIMA does well on MASE but is not as well-calibrated (PCE) as the TSFMs, and neither is N-BEATS.
• Quantile, Student's t and mixture heads are similarly calibrated. These three heads have very similar calibration error across all datasets, indicating no significant advantage among them.
• Gaussian heads are worse. Gaussian prediction heads are significantly worse in calibration and are consistently under-confident, with CCE scores always lower than the other heads. The authors speculate this is due to the limited expressiveness of the Gaussian distribution.
• Shorter forecast horizons harm long-term calibration. For both the trajectory and branching AR methods, models with a shorter forecast horizon H have poorer calibration at a fixed forecast length L; this is more pronounced for the branching method, where horizon lengths of 16 have notably worse PCE than 64 or 128.
• Trajectory AR beats branching AR. The trajectory approach (n = 100 trajectories) has generally lower PCE than the branching method at the same forecast horizon.
• Autoregressive TSFMs are consistently overconfident in long-term forecasting. Both AR methods show that shorter forecast horizons produce more confident forecasts; branching CCE values are often greater than 0.15 for horizon lengths of 16 and 32, and this overconfidence reduces steeply as horizon length increases.
• Non-AR models are efficient and well-calibrated. TiRex and YingLong forecast long-term natively without AR, are significantly more efficient than both AR methods, tend to be better calibrated, and were not significantly over- or under-confident.

Methodology in Plain English

The researchers took five pretrained time series foundation models and used them in a zero-shot setting (no per-dataset training) to forecast six univariate datasets: Reviews, Shopping (M5), Glucose, Heart-Rate, Crime and Patents, covering hourly, daily, 5-minute, second and monthly granularities. For comparison, they trained ARIMA (via AutoARIMA) and N-BEATS (via grid search) on a training portion of each dataset.

Instead of judging calibration with CRPS, WQL or MSIS — which they argue mix calibration with sharpness and accuracy — they used three dedicated measures. PCE compares the predicted quantiles with the fraction of observations that fall below them across q ∈ {0.1, ..., 0.9}. CCE checks whether a predicted interval contains the observed value at the claimed rate across confidence levels s ∈ {0.2, 0.4, 0.6, 0.8}; positive values mean overconfidence and negative values mean underconfidence. SIW measures how wide the predicted intervals are, as a proxy for sharpness. MASE is reported separately for point accuracy of the median forecast.

To isolate the effect of prediction heads, the authors took the latent representation from each pretrained backbone and trained new heads on the TSMixup dataset (chosen because it is independent of the six evaluation datasets), specifically a Gaussian head, a Student's t head, a mixture head (Gaussian, Student's t, log-normal and Laplace) and a retrained quantile head as a control. For long-horizon experiments they compared autoregressive strategies — naive (context extended by the median or mean), branching (Moirai 2.0, separate contexts per quantile) and trajectory (samples from the predicted density used to build n independent paths) — and varied the forecast horizon H. Short-term experiments used prediction length 64; long-term experiments used 256. Context sizes were 512 for most datasets and 128 for Heart-Rate and Patents, with strides d ∈ {1, 4, 8}.

Why This Matters

Calibration determines whether the uncertainty a model reports can be trusted, which matters whenever forecasts feed into decisions rather than just being reported. Prior claims that TSFMs are "well-calibrated" rested largely on CRPS, WQL and MSIS, which this paper argues conflate calibration with sharpness; by measuring calibration directly, the study provides a firmer basis for trusting (or not trusting) TSFM uncertainty estimates.

Real-world applications:

• Healthcare monitoring: calibrated intervals for physiological signals such as glucose and heart rate support risk-aware clinical decisions.
• Energy and demand forecasting: interval forecasts inform grid balancing and capacity planning, where overconfidence is costly.
• Anomaly detection: calibration-aware predicted intervals make it possible to flag observations that fall outside expected ranges at a known rate.
• Retail and consumer behavior modeling: probabilistic demand forecasts (e.g., M5/Shopping-style data) drive inventory and staffing decisions.

Industry relevance: practitioners can now make an informed choice about whether to deploy a TSFM's distributional output as-is, how much to worry about the prediction head design, and how to configure long-horizon autoregressive forecasting. The findings that quantile, Student's t and mixture heads behave similarly, but Gaussian heads do not, and that trajectory AR outperforms branching AR, give concrete guidance for system design.

Future Directions

• Extending the study to fine-tuned models (rather than zero-shot only), to multivariate time series, and to higher-resolution quantile grids beyond the fixed set q ∈ {0.1, ..., 0.9}.
• Investigating how distribution shift and non-stationarity affect calibration in both TSFMs and baselines, given the known sensitivity of deep classification models to distribution shift.
• Further work on long-term forecasting architectures: the authors argue future work should prioritize models with longer forecast horizons and alternatives to autoregressive approaches, since non-AR models like TiRex and YingLong were more efficient and better calibrated.
• Understanding why Gaussian prediction heads are consistently under-confident, and whether more expressive heads can be made even better calibrated without overfitting.

Target Audience

Researchers and practitioners working on time series forecasting, probabilistic prediction and uncertainty quantification; machine learning engineers evaluating whether to deploy time series foundation models in decision-critical pipelines; and anyone who needs to interpret model-reported prediction intervals responsibly. Readers should be comfortable with quantiles, prediction intervals and standard forecasting error metrics, though no deep statistical background is required.

Authors’ abstract

The recent development of foundation models for time series data has generated considerable interest in using such models across a variety of applications. Although foundation models achieve state-of-the-art predictive performance, their calibration properties remain relatively underexplored, despite the fact that calibration can be critical for many practical applications. In this paper, we investigate the calibration-related properties of five recent time series foundation models and two competitive baselines. We perform a series of systematic evaluations assessing model calibration (i.e., over- or under-confidence), effects of varying prediction heads, and calibration under long-term autoregressive forecasting. We find that time series foundation models are consistently better calibrated than baseline models and tend not to be either systematically over- or under-confident, in contrast to the overconfidence often seen in other deep learning models.