Forecast Collapse in Time-Series Foundation Models

AI summary

Overview

Research area: Time-series forecasting with foundation models, focused on financial forecasting and evaluation methodology. Technical level: Intermediate to Advanced — assumes familiarity with forecasting objectives, loss functions, and cross-sectional vs. per-series evaluation. Scope: The paper documents a failure mode it calls "forecast collapse" in time-series foundation models, traces it to target predictability and per-series training objectives, and proposes an objective that trades off calibration against ranking.

What This Paper Is About

When forecasting hourly returns for 1,000 US equities, the authors found that model predictions go nearly flat and rank stocks poorly. They call this "forecast collapse" and set out to determine when and why it happens across time-series foundation models and conventional deep-learning forecasters. Their goal is both diagnostic — explaining the cause — and prescriptive: a training objective that avoids the failure.

Key Contributions

1. Identification of forecast collapse. The authors name and characterize a phenomenon in which forecasts for a large cross-section of equities become nearly flat with poor stock ranking, as measured by cross-sectional correlation.
2. A broad empirical investigation. They examine the phenomenon across time-series foundation models, twelve deep-learning forecasting models, and 97 public benchmark configurations, and link it to target predictability.
3. A two-part explanation and a named tradeoff. They attribute collapse to low predictability capping the amplitude of calibrated point forecasts, and to per-series objectives leaving cross-series structure unidentified. They frame the consequence as a calibration-ranking tradeoff.
4. CalibRank. A simple objective intended to balance calibration and ranking, reported to nearly triple cross-sectional correlation on Finance1K while keeping forecast amplitude close to the target, and to improve correlation across all tested models.

Main Findings

• Collapse is task-dependent: Forecasts for hourly equity returns became nearly flat with poor cross-sectional ranking, but the phenomenon largely disappeared when forecasting trading volume under the same setting.
• Collapse tracks predictability: Across time-series foundation models, twelve deep-learning forecasters, and 97 public benchmark configurations, the phenomenon is closely tied to how predictable the target is.
• Two distinct mechanisms: Low predictability limits how large a calibrated point forecast can be, producing flatness; separately, per-series training objectives never identify structure that spans series, producing weak ranking.
• A calibration-ranking tradeoff exists: Optimizing squared error yields flat predictions, while directly optimizing cross-sectional correlation improves ranking but can inflate forecast amplitude by more than an order of magnitude.
• CalibRank reduces the tradeoff: On Finance1K, the proposed objective nearly triples cross-sectional correlation while keeping amplitude close to the target, and improves correlation on every tested model.
• Evaluation has a blind spot: Conventional per-series metrics can conceal failures in cross-series structure that downstream decisions depend on.

Methodology in Plain English

The authors start from an observed anomaly in an equity return forecasting setup and probe it systematically rather than treating it as an implementation bug. They reproduce the setting with a different target (trading volume) to check whether the failure is tied to the data or to the target itself. They then sweep across time-series foundation models, a set of twelve deep-learning forecasters, and a large collection of public benchmark configurations to see how widely the pattern appears, and relate its severity to how predictable each target is. From that evidence they separate the problem into two causes — amplitude shrinkage under squared-error calibration and missing cross-series structure under per-series objectives — and design CalibRank as a single objective that addresses both. They evaluate it on Finance1K and across all models tested. The abstract does not report the full experimental protocol, model-by-model results, or the exact form of the CalibRank objective.

Why This Matters

This work reframes a familiar symptom — forecasts that look "safe" and flat — as a measurable structural failure rather than acceptable conservatism. It argues that the standard evaluation toolkit for time series can certify a model that is useless for any decision requiring a ranking across many correlated series.

Real-world applications:

• Quantitative investing: Portfolio construction, long-short signals, and risk models depend on ranking thousands of assets, not on each asset's individual error.
• Demand and inventory planning: Ranking SKUs or regions by expected demand drives allocation decisions where cross-item structure matters more than per-item accuracy.
• Energy and load forecasting: Ranking or dispatching across many meters, feeders, or markets requires cross-series discrimination.
• Adoption of time-series foundation models: Teams considering pretrained forecasting models need to know that strong per-series metrics do not imply usable cross-sectional behavior.

Industry relevance is direct: the paper concerns a common production setup (many related series forecast at once), names a failure mode that per-series dashboards would hide, and offers an objective-level remedy rather than a post-hoc rescaling.

Future Directions

• Generalizing beyond finance: The abstract only demonstrates CalibRank on Finance1K; whether it helps in other large cross-sectional settings (retail, energy, traffic) is untested here.
• Formalizing the tradeoff: A theoretical account of why low predictability caps calibrated forecast amplitude, and where the calibration-ranking frontier lies, remains open.
• Rethinking evaluation: If per-series metrics hide cross-series failures, the field needs standard cross-sectional metrics — the paper raises the problem without specifying a full replacement suite.
• Broader objective design: How CalibRank interacts with probabilistic forecasts, multiple horizons, and other model classes is not addressed in the abstract.

Target Audience

Researchers and practitioners working on time-series foundation models, financial machine learning, and large-scale multi-series forecasting; plus evaluation-focused researchers and anyone responsible for choosing forecasting metrics that must support downstream ranking or allocation decisions. Readers unfamiliar with forecasting losses will still follow the core argument, but the calibration-ranking discussion assumes some modeling background.

Authors’ abstract

When forecasting hourly returns for 1,000 US equities, we observe an unexpected phenomenon: predictions become nearly flat and show poor stock ranking, as measured by cross-sectional correlation. We call this forecast collapse. Surprisingly, the phenomenon largely disappears when forecasting trading volume under the same setting. We investigate forecast collapse across time-series foundation models (TSFMs), twelve deep-learning forecasting models, and 97 public benchmark configurations, and find that it is closely tied to target predictability. We identify two distinct reasons behind it: low predictability limits the amplitude of calibrated point forecasts, while per-series objectives leave cross-series structure unidentified. These findings reveal a calibration-ranking tradeoff: optimizing squared error leads to flat predictions, whereas directly optimizing cross-sectional correlation improves ranking but can inflate forecast amplitude by more than an order of magnitude. To address this tradeoff, we introduce CalibRank, a simple objective that balances calibration and ranking. On Finance1K, CalibRank nearly triples cross-sectional correlation while keeping amplitude close to the target, and improves correlation on all tested models. Our results reveal a blind spot in conventional time-series evaluation: per-series metrics can hide failures in cross-series structure needed by downstream decisions.