It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks

AI summary

Overview

Research area: Machine Learning, specifically time series forecasting and the evaluation of time series foundation models (TSFMs).

Technical level: Intermediate.

Scope: This paper introduces TIME, a task-centric benchmark of 50 fresh datasets and 98 forecasting tasks with a pattern-level evaluation scheme, used to evaluate 12 TSFMs under strict zero-shot conditions.

What This Paper Is About

Existing benchmarks for time series foundation models reuse legacy datasets that may already be in model pre-training corpora, apply inconsistent quality control, and impose one-size-fits-all prediction horizons that ignore real deployment contexts. TIME addresses this by building a contamination-free benchmark from freshly curated data, formulating each forecasting task around the operational requirements of its application, and evaluating models by intrinsic temporal pattern rather than by static dataset labels.

Key Contributions

1. TIME benchmark. A task-centric benchmark comprising 50 fresh datasets and 98 forecasting tasks, where prediction configurations are aligned with real-world operational requirements. Using fresh data enables strict zero-shot evaluation free from data leakage.
2. Pattern-level evaluation perspective. A scheme that characterizes each variate by a curated set of interpretable structural time series features with clear separability, enabling pattern-based stratification and retrieval across datasets and yielding generalizable, diagnostic insights.
3. Human-in-the-loop construction pipeline. A pipeline integrating automated screening with human decision-making to finalize dataset structure and to validate task rationality using domain knowledge and LLM-based analysis.
4. Multi-granular analysis and leaderboard. Evaluation of 12 representative TSFMs with an interactive leaderboard supporting multi-granular analysis that combines quantitative results with qualitative prediction visualization. Code is at https://github.com/zqiao11/TIME and the leaderboard at https://huggingface.co/spaces/Real-TSF/TIME-leaderboard.

Main Findings

• Four bottleneck dimensions identified. The authors argue existing benchmarks suffer from legacy-constrained data coverage, compromised data integrity, misaligned task formulations, and limited analysis perspective.
• Metrics can mislead. A TimesFM 2.5 prediction is cited where MASE=0.662 and MAE=1.12 appear favorable, yet the forecast fails to capture the distinct spike structure of the series.
• Top overall performers. Aggregated across all 98 tasks at the task level, Chronos-2, TimesFM-2.5, and TiRex are the top-3, achieving the lowest MASE and CRPS.
• Newer iterations improve. TimesFM-2.5 beats TimesFM 2.0 and 1.0, Moirai2 beats Moirai, and Chronos-2 beats Chronos-bolt, indicating genuine capability gains rather than overfitting to existing benchmark data bias.
• Design trends among top models. Decoder-only frameworks (TimesFM-2.5, Toto, Moirai2) and quantile-based output mechanisms are widely adopted among the top-5.
• Trend patterns. For Trend Strength (F1) and Trend Linearity (F2), most models show larger relative gains over the Seasonal Naive baseline on variates with stronger and more linear trends.
• Seasonality strength (F3). On weakly seasonal variates the performance gap between models is narrow, with most TSFMs clustering around an MASE of 0.7; on strongly seasonal variates model divergence is large, suggesting superior models are those better able to capture strong seasonal signals.
• Seasonality correlation (F4). Early or weaker models (Moirai, VisionTS++, TimesFM-1.0) show a large gap between stable and unstable seasonal patterns; for Chronos-2 and TimesFM-2.5, gains on stable sequences actually surpass those on unstable ones.
• Stationarity (F7). Most TSFMs gain more on non-stationary sequences. Rankings are sensitive to this feature: Toto ranks third on non-stationary data but drops to sixth on stationary sequences, while Chronos-2 leads on stationary sequences but is surpassed by TimesFM-2.5 on non-stationary data. Non-stationary variates account for 16.6% of the total.
• Complexity (F6). Inter-model disparity is smaller on high-complexity variates than on low-complexity ones; high spectral entropy acts as a performance equalizer.
• Aggregation level changes rankings. Pattern-level rankings do not align with the overall task-level rankings; Chronos-2 and TimesFM-2.5 generally remain top two, while Moirai2, TiRex, and Toto fluctuate between third and fifth. The authors state that determining the most appropriate aggregation level remains an open question.
• Conservative forecasts can score well. TSFMs generally predict clear seasonality and trends well, but on high-variation series they tend to produce relatively constant lines that may nevertheless achieve competitive MASE or CRPS, which only visual verification can disambiguate.

Methodology in Plain English

The authors assemble data from four sources: official government portals, industrial and academic partnerships, open-access repositories, and forecasting competitions, deliberately choosing data that existing benchmarks have not used. Each candidate dataset passes a manual eligibility check covering format, context, sequence length, and variable relevance, followed by visualization-based inspection.

An automated screening stage then produces a dataset-level Quality Summary through five sequential steps: filling and aligning timestamps; applying rule-based checks that flag variates with excessive missing rates or near-constant behavior (using top-5 most frequent values or normalized entropy); running a Ljung-Box test to remove white noise; removing extreme outliers with a local interquartile range filter using k=9, replacing detected errors with the preceding valid observation; and computing pairwise correlations to flag highly collinear variates for expert review. Humans then decide whether to prune at the series level or variate level, guided by domain knowledge and LLM-synthesized insights.

Task formulation is context-driven rather than fixed: high-frequency datasets receive three horizons (Short, Medium, Long), while low-frequency or sample-limited datasets get a single operationally viable horizon, with the test length covering complete seasonal cycles.

For evaluation, each variate is decomposed via STL into trend, seasonality, and remainder. Seven structural features are computed: Trend Strength (F1), Trend Linearity (F2), Seasonality Strength (F3), Seasonality Correlation (F4), Residual ACF-1 (F5), Complexity or spectral entropy (F6), and a binary Stationarity indicator (F7) from the Augmented Dickey-Fuller test (stationary if p<0.05). Continuous features are binarized against the population median across all variates; F7 is used directly, producing a 7-bit pattern code. Variates sharing a code can be retrieved for pattern-specific evaluation.

Forecasting uses a rolling window with stride equal to the horizon H, yielding floor(L_test/H) samples per series. Metrics are MASE for point forecasting and CRPS for probabilistic forecasting. Quantiles for distribution-based models are derived by sampling with a default sample size of 100. Scores are normalized against a Seasonal Naive baseline per unit, then aggregated with a geometric mean so that values below 1 indicate better-than-baseline performance.

Why This Matters

Impact on research. The paper argues that plateauing gains on established benchmarks (illustrated by Gift-Eval) make it hard to separate genuine modeling progress from benchmark contamination and evaluation artifacts. By using fresh data and pattern-level analysis, TIME offers a way to test whether reported advances reflect real capability improvements.

Real-world applications.

• Operational forecasting in cloud and IT monitoring, where forecasting horizons must match operational constraints rather than fixed step counts.
• Sales and demand planning, where the valid horizon depends on the business cycle and data frequency.
• Energy and utility load forecasting, where non-stationary behavior and seasonal structure dominate.
• Financial and macroeconomic monitoring, where correlated indicators reflect genuine structural dependencies rather than redundancy.

Industry relevance. The evaluated models come from a range of organizations, and two authors are employed by Datadog, which developed Toto—one of the evaluated models. The paper includes a conflict of interest disclosure stating that evaluations were conducted objectively. The interactive leaderboard and public code support practitioner model selection.

Future Directions

• Determining the most appropriate aggregation level for reporting benchmark results, given that pattern-level and task-level rankings disagree.
• Exploring complex joint patterns across multiple features rather than the single-feature analyses presented, which the authors invite readers to do via the leaderboard.
• Incorporating task-specific metrics tailored to different applications, which the authors state as a plan for future iterations.
• Resolving the ambiguity between statistically good scores and genuinely captured dynamics, since conservative flat forecasts can achieve competitive MASE or CRPS.

Target Audience

Researchers and engineers working on time series foundation models, benchmark designers who need contamination-free evaluation protocols, and practitioners selecting forecasting models for operational deployment. Readers with basic familiarity with forecasting metrics such as MASE and CRPS will benefit most, since the paper assumes comfort with rolling-window evaluation and quantile-based probabilistic forecasting.

Authors’ abstract

Time series foundation models (TSFMs) are revolutionizing the forecasting landscape from specific dataset modeling to generalizable task evaluation. However, we contend that existing benchmarks exhibit common limitations in four dimensions: constrained data composition dominated by reused legacy sources, compromised data integrity lacking rigorous quality assurance, misaligned task formulations detached from real-world contexts, and rigid analysis perspectives that obscure generalizable insights. To bridge these gaps, we introduce TIME, a next-generation task-centric benchmark comprising 50 fresh datasets and 98 forecasting tasks, tailored for strict zero-shot TSFM evaluation free from data leakage. Integrating large language models and human expertise, we establish a human-in-the-loop benchmark construction pipeline to ensure high data integrity and redefine task formulation by aligning forecasting configurations with real-world operational requirements and variate predictability. Furthermore, we propose a novel pattern-level evaluation perspective that moves beyond traditional dataset-level evaluations based on static meta labels. By leveraging structural time series features to characterize intrinsic temporal properties, this approach offers generalizable insights into model capabilities across diverse patterns. We evaluate 12 TSFMs and establish a multi-granular leaderboard to facilitate in-depth analysis and visualized inspection. The leaderboard is available at https://huggingface.co/spaces/Real-TSF/TIME-leaderboard.