Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting

AI summary

Overview

Research area: Deep learning for time series forecasting, specifically the empirical methodology used to benchmark forecasting architectures.

Technical level: Intermediate. The paper is a position paper written in accessible prose, but it assumes familiarity with forecasting concepts such as global versus local models, multivariate versus univariate series, exogenous variables, and spatial (inter-series) versus temporal (intra-series) processing.

Scope: The paper argues that current benchmarking practices in deep learning for time series forecasting fail to identify which design choices actually cause performance differences, and proposes a forecasting model card plus a benchmarking philosophy of isolated, compartmental comparisons.

What This Paper Is About

The number of new forecasting architectures keeps growing, yet published empirical results are often contradictory, so it is hard to know which design choice or model component is actually responsible for performance. This position paper argues that current benchmarking practices overlook fundamental design dimensions, treating them as mere implementation details, and that this slows progress in the field. To support the position, the authors quantify the impact of these overlooked choices on state-of-the-art (SOTA) architectures and show that a streamlined architecture built on well-understood design principles can match current SOTA.

Key Contributions

1. The authors analyze the current state of deep learning for time series forecasting by relying on principles for forecasting groups of time series to explain often contradictory empirical results.
2. They empirically quantify the impact of overlooked design choices and implementation details in SOTA architectures, showing that these explain a significant portion of observed performance improvements.
3. They show that a streamlined architecture built on well-understood design principles can match the performance of the current SOTA.
4. They introduce an auxiliary forecasting model card template, complementary to existing generic model cards (Mitchell et al., 2019), to help designers and practitioners characterize existing and new forecasting architectures along key design choices.

Main Findings

• Four design dimensions frame the analysis. The paper identifies D1 model configuration (local, global, or hybrid), D2 preprocessing and exogenous variables, D3 temporal processing, and D4 spatial processing as the dimensions that most affect forensic comparison of forecasting models.
• Model configuration changes results drastically. In Table 1, comparing architectures with local ("hybrid") versus global parameters for a horizon of 96 produces markedly different MSE. For example, on Traffic the reference Transformer scores 0.417 ± .009 as hybrid versus 0.392 ± .000 as global; Crossformer scores 0.540 ± .014 versus 0.512 ± .007; TimeMixer 0.464 ± .001 versus 0.463 ± .001; and iTransformer 0.435 ± .002 versus 0.409 ± .000. On Solar the pattern reverses in places: Transformer 0.196 ± .000 versus 0.205 ± .001, Crossformer 0.177 ± .008 versus 0.166 ± .005, TimeMixer 0.366 ± .017 versus 0.367 ± .017, iTransformer 0.189 ± .001 versus 0.197 ± .002. The authors note that such local parameters are often discoverable only by inspecting code, citing TimeMixer's learnable local parameters in the normalization module as an example.
• Exogenous variables and preprocessing shift results. Table 2 shows that adding or removing the same covariates (calendar features) changes MSE. PatchTST on Electricity scores 0.128 ± .000 with exogenous variables versus 0.134 ± .000 without; on Traffic 0.355 ± .000 versus 0.383 ± .001; on Weather 0.174 ± .000 versus 0.180 ± .002. iTransformer on Electricity scores 0.154 ± .000 with exogenous variables versus 0.167 ± .000 without. The authors note PatchTST, DLinear, and Crossformer do not use covariates in their original implementations, while iTransformer does.
• No single model consistently wins. In Table 3, the reference architectures (MLP, TCN, RNN, Transformer, Pyraformer-style attention) achieve competitive performance against DLinear, PatchTST, and TimeMixer across all considered scenarios for a horizon of 96, using standardized inputs including covariates and with hyperparameter tuning. On Weather, the local OLS Linear model is among the best performers (MSE 0.144, MAE 0.209), which the authors attribute to Weather being a multivariate series with heterogeneous channels that the local model explicitly treats as such.
• Spatial attention can be removed with no loss, or with gains. Table 4(b) reports an ablation replacing iTransformer's spatial attention layer with a simple MLP: Electricity 0.148 ± .000 versus 0.149 ± .001, Weather 0.171 ± .001 versus 0.171 ± .000, Traffic 0.393 ± .001 versus 0.390 ± .001, and Solar 0.208 ± .003 versus 0.194 ± .001. The authors report that removing spatial attention entirely led to better or similar performance on all considered datasets.
• Simple operators remain competitive under spatial processing too. Table 4(a) compares MLP + spatial attention and Pyraformer-style attention + spatial attention against iTransformer, Crossformer, and ModernTCN, again showing that streamlined architectures perform comparably to SOTA.
• Accuracy must be weighed against computational cost. Figure 1 reports MSE against mean batch time during training on the Electricity dataset for a horizon of 96, with circle size indicating memory consumption, for models with and without spatial processing.
• The proposed remedy. The paper calls for benchmarks that isolate design dimensions and for explicit documentation of design choices, offering the auxiliary forecasting model card (Figure 2) with fields for model setting, D1 model configuration, D2 preprocessing and exogenous variables, D3 temporal processing, and D4 spatial processing.

Methodology in Plain English

The authors take a position rather than proposing a new architecture. They first reframe forecasting of a collection of time series in formal terms, distinguishing local models (parameters specific to a single series), global models (parameters shared across series, and usable inductively on unseen series), and hybrid global-local models that share some parameters but keep series-specific ones. They then select four design dimensions and examine each one in turn.

For empirical evidence, they pick representative SOTA models that shaped recent research: PatchTST, DLinear, TimeMixer, and a simple Linear model trained with L2 regularization and OLS, plus models with spatial processing: iTransformer, ModernTCN, and Crossformer. They compare these against reference streamlined architectures that stack a temporal module (MLP with residual connections, causal dilated TCN, gated RNN, Transformer layers, or pyramidal attention) with an optional spatial attention layer. All models are evaluated under the same benchmarking setup, unified settings, and with access to the same exogenous variables, using the available open-source implementations adapted to a standardized input pipeline.

The benchmarks are four real-world datasets from different domains: Electricity (hourly electricity usage for 321 customers), Weather (21 meteorological variables every 10 minutes from Germany), Traffic (hourly road occupancy from 862 sensors in San Francisco), and Solar (10-minute solar power generation records from 137 photovoltaic plants). Data are split 70%/10%/20% for training, validation, and testing. Metrics are computed on scaled data for consistency with published benchmarks, and results report standard deviations across 3 independent runs with different random seeds. The experiments use an input window of 96 in the main body, with a longer window of 336 in Table 3 (except for Solar).

Rather than ranking models, the authors run targeted manipulations—adding or removing local parameters, adding or removing covariates, and replacing spatial attention with an MLP—to expose how much of a reported gain is attributable to choices unrelated to the sequence modeling operators at the center of most architecture papers.

Why This Matters

The paper's central claim is that benchmarks that compare monolithic architectures on final accuracy cannot attribute performance gains to specific components, so reported improvements may come from factors such as global versus local parameterization, covariate usage, or preprocessing rather than methodological innovation. The authors argue this makes current research "seemingly stuck in a loop" of positive results being quickly dismissed by new evidence. They position better benchmarking as a requirement for the field's maturity.

Real-world applications implicated by the datasets and dimensions analyzed:

• Electricity load forecasting for utilities serving many customers with shared vs. per-customer models.
• Traffic management using large networks of road occupancy sensors.
• Weather forecasting with heterogeneous meteorological channels that may be better handled by local heterogeneous models.
• Solar power generation forecasting across many photovoltaic plants.

Industry relevance: The paper calls for practitioners and researchers to provide all compared methods with the same input data, to avoid combining new experiments with literature results unless an identical evaluation setup can be guaranteed, and to document design choices explicitly. It points to model cards as a way to cut the time and effort needed to extract design information from code or supplementary materials, and to reduce the errors and ambiguities of that exercise. It also notes that increased computational cost from spatial processing should be justified by forecasting accuracy, which matters for deployment decisions.

Future Directions

• Benchmarks that isolate design dimensions, possibly using synthetic datasets, to measure the individual effects of different components rather than only end-to-end accuracy.
• Statistical comparison procedures so that observed performance gains can be shown to be significant.
• Adoption of the auxiliary forecasting model card as a documentation standard, which the authors argue can also guide the design of ablation studies—they note that a missing ablation study is often a consequence of overlooking design aspects that model cards make explicit.
• Handling dimensions that cannot be fully disentangled, since the authors acknowledge that the design dimensions are not always orthogonal—space and time, for example, can be processed in an integrated fashion—and suggest explicitly acknowledging architectural coupling when full separation is not possible.

Target Audience

This paper is most useful for forecasting researchers who design and benchmark new architectures, for practitioners selecting among published forecasting models for deployment, and for benchmark maintainers and reviewers who assess empirical claims in time series forecasting. It is also relevant to anyone interested in research methodology and reproducibility in machine learning more broadly.

Note: The provided paper content is truncated during Section 5 (Alternative Views), so the full list of alternative views and any later sections are not reported here.

Authors’ abstract

Deep learning models have grown popular in time series applications. However, the large quantity of newly proposed architectures and the often contradictory empirical results make it difficult to assess which design choice and model component drives performance. In this position paper, we argue that current benchmarking practices fail to identify the factors responsible for performance differences, thus slowing down progress in the field. In particular, differences in crucial design dimensions are overlooked when comparing architectures, ultimately leading to inconsistent outcomes. To support our position, we show that such differences-often treated as mere implementation details-can have a greater impact than adopting specific sequence modeling layers. We discuss how overlooked aspects (such as globality and locality) can (1) fundamentally change the class of the forecasting method and (2) drastically affect empirical results. Our findings suggest rethinking our benchmarking practices and focusing on the foundational aspects of the forecasting problem when designing and comparing architectures. As a concrete step, we propose an auxiliary forecasting model card, i.e., a template with a set of fields to characterize existing and new forecasting architectures based on key design choices.