SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series

AI summary

Overview

Research area: Time series forecasting evaluation and benchmarking (deep learning for time series, model diagnostic frameworks).

Technical level: Intermediate. The motivation and conclusions are readable without deep math, but the evaluation metrics (MSE/MAE, SNR, MSE_Obs vs. MSE_True), the model families compared, and the theoretical-optimum argument assume some familiarity with forecasting.

Scope (one sentence): The paper introduces SynTSBench, a synthetic-data evaluation framework with programmable feature configuration that isolates individual temporal patterns, injects controlled noise and anomalies, and compares deep forecasting models directly against mathematical optima.

What This Paper Is About

Deep forecasting models often score well on standard benchmarks yet remain unreliable in real deployments, partly because real-world time series bundle trends, seasonality, and cross-variable dependencies together so no one can tell which pattern a model actually learned. Existing benchmark suites (ProbTS, TFB, BasicTS, BasicTS+, TSlib) also report no theoretical performance ceiling, so it is unclear whether a model is generalizing or just fitting noise. SynTSBench addresses this by generating controlled synthetic series where the correct answer is known analytically, letting the authors measure each model's ability pattern by pattern.

Key Contributions

1. A synthetic, programmable evaluation paradigm. SynTSBench builds datasets from configurable primitives — 11 trend functions, 10 periodic patterns, ARMA and random-walk processes, lag and nonlinear cross-variable relationships — so that a model's performance can be attributed to a specific capability rather than to coincidental correlations in real data.

2. Three analytical dimensions. The framework maps (a) temporal feature decomposition and capability mapping, (b) robustness under data irregularities with quantified noise tolerance and anomaly recovery, and (c) theoretical optimum benchmarking that places model predictions against mathematical optima for each pattern type.

3. A broad model comparison. The paper states it evaluates 12 fine-tuned models spanning transformer, MLP, CNN, RNN, and KAN families (Autoformer, PatchTST, iTransformer, CATS, TimeLLM, TimeMixer, TSMixer, PaiFilter, TexFilter, DLinear, N-BEATS, N-HiTS, TimesNet, SegRNN, TimeKAN are named in the text), plus three zero-shot models (Chronos, TimeMoE, Moirai), and compares against traditional baselines (ARIMA, mean, naive, ARMA(1,1)).

4. A public code release at https://github.com/TanQitai/SynTSBench.

Main Findings

• No universal winner. Across all dimensions, no single architecture achieves optimal performance; the paper reports that current deep learning models do not universally approach the optimal baselines across all types of temporal features.

• Trend learning favors MLP architectures. For trend forecasting, patterns with accelerating growth are hardest. PaiFilter ranks first on 5 of the 11 trend patterns, followed by TexFilter, N-BEATS, and DLinear. Among non-MLP models, only PatchTST (transformer-based) took first place once.

• DLinear dominates pure periodic signals. In the periodic tests, DLinear achieves the best result on 7 of 10 functions. All models struggle with Square-Wave patterns, which contain discontinuous jumps.

• Observation-based error hides model differences under noise. At the extreme noise level (SNR = -10 dB), MSE_Obs across most models converges to approximately 1 and becomes useless for ranking. MSE_True retains discriminative power, showing large differences between architectures — for example, PaiFilter tracks the true underlying signal while Autoformer fails to recover structure and N-HiTS visibly overfits the noise.

• TimesNet leads under extreme noise, transformers degrade most. TimesNet performs best at extreme noise, closely followed by the FilterNet models (PaiFilter and TexFilter); transformer-based architectures show the largest degradation as noise increases.

• Anomaly resilience varies sharply. N-BEATS and TimeKAN show negligible degradation under anomalies, and TimeMixer deteriorates by only about 5 percent. PaiFilter, TexFilter, and TimeKAN keep relatively stable error rates even at 10 percent point anomalies. N-HiTS shows the worst robustness with catastrophic collapse (its extreme degradation is excluded from the visualization), and TSMixer shows a 342 percent error increase as anomaly density rises.

• Traditional statistics still leads on standard ARMA. For ARMA(1,1) and ARMA(2,2), ARIMA retains a slight edge; DLinear and TimeKAN come closest among deep models. On the dedicated lag-50 ARMA-Long-Dependency test, DLinear and TimeKAN approach ARIMA, while Autoformer underperforms despite its auto-correlation mechanism and SegRNN's slice encoding fails to capture the sparse long-range relationships.

• Random walks reward the naive baseline. SegRNN and TimeKAN nearly match the theoretical optimum (naive forecast) on random-walk data, whereas TSMixer's MSE is 15 times higher than the best model.

• Channel dependence helps multivariate tasks. On lag relationships, channel-dependent models (TimesNet, TSMixer, TimeMixer) predict the lagged variable with roughly 50 percent lower MSE than the source variable, while channel-independent models (PaiFilter, TexFilter, PatchTST) show no such gap. TSMixer and TimesNet consistently lead the multivariate scenarios, and in the five-variable complex scenario TimeMixer and TimesNet reach near-optimal performance.

• Strength profiles differ by capability. TimeKAN, TimeMixer, PaiFilter, TexFilter, and TimesNet perform strongly overall, but TimeKAN, PaiFilter, and TexFilter struggle with cross-variable relationships and TimesNet is weak at short-distance dependency modeling. N-HiTS, Autoformer, and SegRNN show significant weaknesses across many areas.

Methodology in Plain English

Instead of testing models on messy real data, the authors generate time series whose structure they control completely. Because they build each series from known mathematical components (functions for trend, sine waves for seasonality, ARMA processes for autocorrelation, engineered lag and nonlinear links between variables), they can compute what a perfect forecast would look like — the "theoretical optimum" — and compare each model's prediction against it.

They feed these clean series to models through one fixed protocol: input length 96, forecast horizons of 10, 24, 48, 96, and 192, with 7:1:2 train/validation/test splits for deep learning models and 8:2 for traditional methods. To test robustness, they add controlled Gaussian noise at target signal-to-noise ratios ranging from clean data down to -10 dB, add other noise types (uniform, Laplace, t-distribution, Lévy stable), and inject anomalies — point anomalies at rates from 0.5 percent to 10 percent, pulse anomalies in counts of 1, 3, and 5, plus mean shifts and trend shifts. Crucially, they score predictions two ways: against the noisy observations (MSE_Obs) and against the hidden clean signal (MSE_True), which is what exposes models that quietly memorize noise.

Why This Matters

Research impact. The paper supplies something real benchmarks cannot: known ground truth and per-pattern attribution. This turns model comparison from a leaderboard ranking into a diagnosis, and it gives researchers a way to tell genuine generalization apart from noise overfitting.

Real-world applications (drawn from the scenarios the paper names or simulates):

• Financial market trend analysis, where distinguishing learned structure from coincidental correlation carries direct cost.
• Energy system dispatch optimization, where forecasts feed operational decisions.
• Climate pattern forecasting, where long-horizon trend and seasonal behavior both matter.
• Urban traffic prediction, which mixes daily (24h), weekly (7d), and seasonal cycles; the paper also builds synthetic datasets for economic indicators, coupled dynamic systems, and weather-sales relationships.

Industry relevance. Model selection currently resembles what the authors call "blind-box" parameter tuning. Quantified noise tolerance thresholds, anomaly recovery behavior, and distance from a theoretical ceiling give practitioners concrete criteria for choosing an architecture for a specific data regime rather than defaulting to the latest benchmark winner.

Future Directions

• Improve universality. The authors suggest developing architectures with better generalization and adaptability across diverse temporal patterns, since every tested model has blind spots.
• Close the gap to theoretical optima. Because the framework establishes per-pattern performance boundaries, the natural next step is targeted architectural work on the specific patterns where models fall short — accelerating growth trends, discontinuous square waves, sparse long-range dependencies, and cross-variable relationships.
• Exploit the robustness findings. Transformer degradation under heavy noise and the collapse of N-HiTS under anomalies are concrete failure modes that motivate new noise- and anomaly-aware designs.
• Extend beyond fine-tuning. The paper includes zero-shot models (Chronos, TimeMoE, Moirai) but notes that state-of-the-art models still struggle with zero-shot generalization, leaving broader zero-shot evaluation as an open direction. Additional threads the authors ran but leave for further reading include dataset-scale sensitivity, synthetic real-world simulations, and limitations of batch normalization.

Target Audience

Forecasting researchers and benchmark designers who need diagnostic rather than aggregate evaluation; practitioners in finance, energy, climate, and traffic forecasting who must pick models for specific pattern regimes; and graduate students or engineers building or auditing time series architectures, who will find the capability map, the noise-tolerance results, and the released code directly usable.

Authors’ abstract

Recent advances in deep learning have driven rapid progress in time series forecasting, yet many state-of-the-art models continue to struggle with robust performance in real-world applications, even when they achieve strong results on standard benchmark datasets. This persistent gap can be attributed to the black-box nature of deep learning architectures and the inherent limitations of current evaluation frameworks, which frequently lack the capacity to provide clear, quantitative insights into the specific strengths and weaknesses of different models, thereby complicating the selection of appropriate models for particular forecasting scenarios. To address these issues, we propose a synthetic data-driven evaluation paradigm, SynTSBench, that systematically assesses fundamental modeling capabilities of time series forecasting models through programmable feature configuration. Our framework isolates confounding factors and establishes an interpretable evaluation system with three core analytical dimensions: (1) temporal feature decomposition and capability mapping, which enables systematic evaluation of model capacities to learn specific pattern types; (2) robustness analysis under data irregularities, which quantifies noise tolerance thresholds and anomaly recovery capabilities; and (3) theoretical optimum benchmarking, which establishes performance boundaries for each pattern type-enabling direct comparison between model predictions and mathematical optima. Our experiments show that current deep learning models do not universally approach optimal baselines across all types of temporal features.The code is available at https://github.com/TanQitai/SynTSBench