Statistical benchmarking of transformer models in low signal-to-noise time-series forecasting

AI summary

Overview

Research area: Machine learning for multivariate time-series forecasting, with a focus on transformer architectures, synthetic benchmarking, and attention sparsity.

Technical level: Intermediate. The paper combines accessible experimental framing (benchmark tables, effect types) with technical material on attention mechanics, a dynamic sparsification algorithm, and an analytical derivation of OLS performance in Appendix B.

Scope: The paper builds a controlled synthetic framework with tunable signal-to-noise ratios and dependency structures to test whether two-way attention transformers beat classical baselines, and introduces a data-driven attention sparsification method for noisy regimes (Cyril Garcia, Guillaume Remy; arXiv:2602.09869v1 [cs.LG], 10 Feb 2026).

What This Paper Is About

Transformer models have been proposed widely for time-series forecasting, but recent work questions whether they genuinely beat simpler linear models on standard benchmarks. The authors argue that fixed real-world benchmark datasets make it hard to isolate why and when transformers help, so they generate synthetic data with known ground-truth dependency structures and controlled noise. Their goal is to statistically evaluate transformers against Lasso, boosting, and multilayer perceptrons, and to test a dynamic attention sparsification procedure for low signal-to-noise settings.

Key Contributions

1. A controlled synthetic benchmarking framework for multivariate forecasting with tunable "orders" of dependency (Order 0: linear; Order 1: temporal/cross-sectional shifts and feature non-linearities; Order 2: combined shifts and non-linear interactions in the time-series or cross-sectional dimension) and systematically varied noise, evaluated via out-of-sample correlation against the optimal ground-truth predictor.

2. Benchmarking of two-way attention transformers — architectures that alternate temporal and cross-sectional self-attention blocks (denoted by strings such as "TCTC") — against Lasso regression, boosting, OLS, and a fully connected MLP baseline.

3. A dynamic sparsification procedure for attention matrices, applied during training. Attention rows are treated as discrete probability distributions, and entries smaller than a fraction K of the row maximum are removed. The authors report in the introduction that this yields up to 10–20% performance gains in correlation with the optimal predictor.

4. An analytical computation in Appendix B giving the expected out-of-sample correlation for OLS in the linear case, bridging empirical results with statistical theory.

Main Findings

• Linear effect (Order 0, Table 1): Lasso is the best model, as expected for a linear relationship. The transformer performs "quite well at small ρ" relative to Boosting and MLP. At ρ = 0.50, Lasso reaches 0.995 while Trans reaches 0.693.

• TS-Shift effect (Table 2): Lasso again performs well. The transformer reaches 0.734 at ρ = 0.50, but the paper notes Trans "does not perform quite as well on TS-Shift as on the Linear effect."

• CS-Shift effect (Table 3): The transformer performs strongly, with 0.589 at ρ = 0.50 versus Lasso's 0.435 and MLP's 0.419. The authors state cross-sectional attention detects this shift well relative to benchmarks.

• Fea-Nonlin effect (Table 4): Lasso fails (values near zero or negative at every ρ), Boosting and MLP fail, while the transformer is the only model that detects the non-linear feature interaction, reaching 0.493 at ρ = 0.50.

• TSCS-Shift effect (Table 5): The transformer performs poorly (0.029 at ρ = 0.50), while MLP is best (0.433). The authors leave understanding this failure to future work.

• Superposition of all five effects (Tables 6 and 7): At ρ = 0.05, Lasso has the best correlation to the optimal predictor (0.103) versus Trans (0.065). At ρ = 0.20, Lasso reaches 0.605 versus Trans 0.225. However, Trans shows more balanced correlation across individual effects — at ρ = 0.20 it reaches 0.118 on Fea-Nonlin (Lasso: 0.021) and 0.102 on CS-Shift (Lasso: 0.084).

• Overall transformer profile: The paper concludes the transformer "strikes a good balance" — decent on linear and TS-Shift effects, the only model detecting non-linear feature interactions, and on average the best on CS-Shift, with TSCS-Shift as its shortcoming.

• Dynamic sparsification on lag-1 TS-Shift (Table 8): The deterministically sparse model informed by ground truth gives the best results (0.096, 0.200, 0.469 across ρ = 0.015, 0.03, 0.1). The dynamic max_sparse model beats full_attention at ρ = 0.015 (0.025 vs 0.017) and ρ = 0.03 (0.083 vs 0.071). The p-value is 0.028 at ρ = 0.015 and rises monotonically with ρ (0.126 at 0.03, 0.467 at 0.1).

• Dynamic sparsification on CS-Shift (Table 9): max_sparse improves over full_attention at ρ = 0.015 (0.019 vs 0.012, p-value 0.021) and ρ = 0.03 (0.038 vs 0.034, p-value 0.094).

• Dynamic sparsification is neutral when sparsity is not optimal: On the superposition of effects (Table 10) and the Linear effect (Table 11), max_sparse remains competitive with full_attention, with high p-values (0.638, 0.464, 0.953 for superposition; 0.902, 0.779, 0.724 for Linear), indicating no significant difference.

• Learned attention structure (Appendix A): At ρ = 0.03, the temporal attention matrix shows high probability mass along the sub-diagonal, and the cross-sectional matrix often shows one dominant coefficient per row — consistent with the sparsity intuition, though the cross-sectional matrix would have exactly one non-zero coefficient per row absent noise.

Methodology in Plain English

The researchers deliberately avoid real-world datasets. Instead, they generate synthetic data where the ground truth is fully known, so they can measure how close a model's predictions come to the theoretically optimal predictor.

Data generation. Predictors X are sampled i.i.d. from a standard normal distribution. A "true" target signal is built by combining one or several canonical effects — linear combinations, shifts in time or across series, non-linear feature interactions, and combinations thereof. Noise is then added so that the resulting target Y has a precisely controlled correlation ρ with the ideal signal: Y = Ỹ + √(1−ρ²)·Z with Z standard normal. Values of ρ as low as 2% (0.02) are tested.

Default dimensions. Unless stated otherwise: T_train = 2500, T_test = 1500, N = 10 series, F = 20 features, and a lookback window T_win = 10. The authors note these dimensions correspond in order of magnitude to roughly ten years of daily observations.

Baselines. OLS fitted per time series (for which they derive an expected correlation formula C(ρ, γ) with γ = T_win·N·F / T_train), a global Lasso with cross-validated penalty, a global boosting model, and a global MLP that flattens all inputs into a T_win × N × F vector and applies four 512-dimensional linear layers with GELU activations and 0.1 dropout.

Transformer. Features are projected to d_model = 64; two learned positional embeddings are added (one for time, one for series). Alternating blocks perform temporal attention (over T_win, separately per series) and cross-sectional attention (over the N series, per time step), each with 8 attention heads and a feed-forward block of internal dimension 256, with 0.1 dropout.

Dynamic sparsification. Attention probabilities are averaged over the batch to get matrix P̄. For each row, the maximum entry M is found, and any entry below K·M (with K = 0.1 in all experiments) is masked out via a −∞ bias before the final softmax. The authors considered top-k and quantile-based alternatives but rejected them because they impose a fixed sparsity level and behave unevenly under causal masking.

Statistical testing. For the sparsity experiments, they run bootstraps (n = 90 samples) in which the dataset is fully regenerated with identical statistical properties, and compare means of out-of-sample correlations using a two-sample hypothesis test reporting a p-value.

Why This Matters

Impact on research. The paper responds directly to skepticism about transformers in time-series forecasting, and it argues that fixed benchmark datasets obscure which architectural features actually help. By using synthetic processes with known structure, it isolates specific dependency types where transformers win (cross-sectional shifts, non-linear feature interactions) and where they lose (combined temporal–cross-sectional shifts, pure linear effects). The analogy drawn between attention sparsity and sparsity-inducing regularization in classical regression offers a mechanistic hypothesis for why these models generalize under noise.

Real-world applications mentioned or implied by the setup:

• Finance — forecasting from multivariate market series in noisy environments.
• Meteorology — the paper's own example: predicting next-day temperature across cities, with time indexing days, series indexing locations, and features covering temperature, wind, and humidity.
• Supply chain management — data-driven decision-making cited in the introduction.
• Energy management — a further domain cited as a cornerstone of time-series forecasting.

Industry relevance. The paper targets settings with limited data — a few years of daily observations — which is common in practice and much smaller than typical deep learning benchmark datasets. The reported finding that sparsification helps specifically when ρ is on the order of a few percent is framed as relevant to genuinely low signal-to-noise problems, and the sparsification mechanism is compatible with existing attention layers. A GitHub repository is provided at https://github.com/cyrilgarcia009/TSNN.

Future Directions

1. Understanding TSCS-Shift failure. The transformer failed to detect effects that shift simultaneously in the time-series and cross-sectional directions, even with multiple stacked attention layers. The authors explicitly leave this open and call for research on how attention dimensions should be composed or integrated.

2. Extending sparsification to deeper and multi-head settings. The sparsity analysis focuses on a single temporal layer plus a single cross-sectional layer ("TC") with one head per layer for visualization purposes, though the authors state the mechanism naturally extends to deeper architectures and multi-head attention — extensions they do not explore.

3. Learning or tuning the threshold K. K is fixed at 0.1 in all experiments; the paper notes it could be learned during training or selected via cross-validation, which is not done here.

4. Preprocessing and non-stationarity. The framework assumes stationary, preprocessed, normalized predictors, and the authors explicitly defer the preprocessing stage to a later study — leaving open how these results carry over to non-stationary real-world series.

Target Audience

Researchers and practitioners working on time-series forecasting with deep learning, particularly those evaluating whether transformer architectures are justified over simpler linear or boosting baselines. Also relevant to machine learning researchers interested in benchmarking methodology, synthetic data design for statistical significance testing, and sparse or structured attention mechanisms. Readers need some familiarity with attention, regression baselines, and correlation-based evaluation to follow the tables and the Appendix B derivation.

Authors’ abstract

We study the performance of transformer architectures for multivariate time-series forecasting in low-data regimes consisting of only a few years of daily observations. Using synthetically generated processes with known temporal and cross-sectional dependency structures and varying signal-to-noise ratios, we conduct bootstrapped experiments that enable direct evaluation via out-of-sample correlations with the optimal ground-truth predictor. We show that two-way attention transformers, which alternate between temporal and cross-sectional self-attention, can outperform standard baselines-Lasso, boosting methods, and fully connected multilayer perceptrons-across a wide range of settings, including low signal-to-noise regimes. We further introduce a dynamic sparsification procedure for attention matrices applied during training, and demonstrate that it becomes significantly effective in noisy environments, where the correlation between the target variable and the optimal predictor is on the order of a few percent. Analysis of the learned attention patterns reveals interpretable structure and suggests connections to sparsity-inducing regularization in classical regression, providing insight into why these models generalize effectively under noise.