SimDiff: Simpler Yet Better Diffusion Model for Time Series Point Forecasting

AI summary

Overview

Research area: Time series forecasting with diffusion models (generative deep learning for multivariate point and probabilistic forecasting).

Technical level: Advanced. The paper assumes familiarity with denoising diffusion probabilistic models (DDPMs), Transformer architectures, and forecasting benchmarks such as ETTh1, ETTm1, Weather, Traffic and Electricity.

Scope: The paper proposes SimDiff, a single-stage end-to-end Transformer diffusion model for time series point forecasting, and evaluates it against diffusion-based and regression-based baselines on CRPS, CRPS-sum, MSE and inference time.

What This Paper Is About

Diffusion models produce diverse probabilistic forecasts but have generally failed to match regression-based methods at point estimation, because they struggle to inject enough contextual bias from past observations and to balance sample diversity against the stability and precision that point forecasts need. SimDiff addresses this by using one unified Transformer as both denoiser and forecaster—no pre-trained or jointly trained regressor attached—and by extracting accurate point forecasts from the model's own probabilistic samples via a Median-of-Means ensemble. The goal is a simpler diffusion model that nonetheless achieves state-of-the-art point accuracy.

Key Contributions

1. A fully end-to-end diffusion forecaster. SimDiff is presented as the first fully end-to-end diffusion model achieving stable state-of-the-art results in time series point forecasting, using a single unified network as both denoiser and forecaster and eliminating external pre-trained or jointly trained predictors.

2. Normalization Independence (N.I.). A diffusion-specific normalization scheme in which past samples are instance-normalized and rescaled by learnable affine parameters, while future targets are normalized with their own statistics independently, and only during training. This is designed to better capture data distributions and mitigate temporal drift.

3. A streamlined Transformer denoising backbone. A patch-based tokenization scheme with Rotary Position Embedding (RoPE), channel independence, and no skip connections—removing the U-Net-style skip connections (as used in U-ViT) that the authors argue can amplify noise in time series.

4. A Median-of-Means (MoM) estimator for point forecasting. The MoM estimator aggregates probabilistic samples—splitting data of size n into K subsamples of size B, taking the median of their means, repeating R times with shuffled data, and averaging the R medians—to convert distributional samples into a stable point estimate and deliver state-of-the-art point results.

Main Findings

• Strong probabilistic performance without explicit probabilistic optimization. On CRPS and CRPS-Sum (multivariate setting), SimDiff records 0.22 / 0.019 on ELEC, 0.16 / 0.039 on TRAFFIC, 0.42 / 0.166 on TAXI, and 0.41 / 0.057 on WIKI, competitive with leading probabilistic models despite not being explicitly optimized for that task.

• Best MSE on 6 of 9 datasets. In Table 2, SimDiff achieves rank 1 on NorPool (0.534), Electricity (0.145), Exchange (0.015), ETTh1 (0.394), ETTm1 (0.322) and Wind (0.880), and rank 2 on Caiso (0.106), Traffic (0.383) and Weather (0.299). Its average rank is 1.33, the best among all listed methods; results are averaged over 5 runs with variance under 1e-5.

• Average 8.3% MSE reduction over other diffusion models. SimDiff reduces MSE by an average of 8.3% across all datasets compared to diffusion models such as mr-Diff.

• Ensembling improves the single-shot model. Moving from one inference to the MoM ensemble reduces MSE from 0.408 to 0.394 on ETTh1, 0.317 to 0.299 on Weather, 0.901 to 0.880 on Wind, and 0.110 to 0.106 on Caiso. MoM outperforms simple averaging (0.398, 0.305, 0.887, 0.109) on all four datasets.

• Normalization Independence consistently helps. Removing N.I. worsens MSE on every tested dataset: NorPool 0.534→0.555, Electricity 0.145→0.151, Traffic 0.383→0.389, ETTm1 0.322→0.327, Weather 0.299→0.328, Wind 0.880→0.891, Exchange 0.015→0.019, with the largest degradations on datasets with severe out-of-distribution drift (Weather, NorPool).

• Sample variance is low. Table 3 reports sample variance averaged over features and horizons for SimDiff of 0.012 (ETTh1), 0.012 (ETTm1), 0.005 (Weather), 0.011 (NorPool) and 0.018 (Wind), alongside lower single-shot MSE than the compared diffusion baselines.

• Large inference speed advantage. On ETTh1 single inference, SimDiff takes 0.22 ms (H=96), 0.24 ms (H=168), 0.30 ms (H=192), 0.33 ms (H=336) and 0.46 ms (H=720), versus 4.73–8.40 ms for TimeDiff, 7.02–10.92 ms for mr-Diff, 67.02–379.80 ms for CSDI, 111.23–483.39 ms for TMDM, 135.92–886.56 ms for SSSD and 294.85–2312.26 ms for TimeGrad—over 90% improvement in inference speed.

• A weighted MAE loss replaces the usual objectives. The denoising objective is a weighted mean absolute error scaled by the cumulative noise term, rather than a likelihood-based objective.

Methodology in Plain English

SimDiff frames forecasting conventionally: given L past multivariate observations of dimension M, predict the next H values. It then applies the standard conditional DDPM machinery—corrupting the future window with Gaussian noise in a forward process, and training a network to reverse that corruption step by step, conditioned on the past.

The distinctive choices are in the conditioning and the backbone. Instead of normalizing past and future windows with the same statistics (which assumes stationarity), the past is normalized with its own statistics and a learnable affine layer, while the future target is normalized by its own statistics during training only; at test time, predictions start from standard Gaussian noise and are de-normalized using past statistics plus the learned affine parameters. This decoupling is intended to prevent bias when the future shifts in level or scale.

The denoiser is a plain Transformer rather than a U-Net: the series is split into overlapping patches that become tokens, a diffusion timestep is turned into an additional token, RoPE encodes relative position, channels are processed independently, and skip connections are removed. Because the model is trained end to end on the future series, it is simultaneously the denoiser and the forecaster.

At inference, the model generates many sample trajectories; the MoM estimator then combines them into a single point forecast by dividing the samples into groups, taking group means, taking the median of those means, repeating this over shuffled groupings, and averaging the resulting medians. The authors compare this against simple averaging and single-sample inference.

Why This Matters

Impact on research. The paper challenges a prevailing design pattern in diffusion-based forecasting—attaching a pre-trained or jointly trained regressor to supply contextual bias (as in TimeDiff, mr-Diff and TMDM). It argues that this stabilizes optimization at the cost of generative flexibility, sample coverage and maintenance overhead, and shows that a single end-to-end Transformer with tailored normalization can instead achieve state-of-the-art point accuracy while retaining the samples needed for probabilistic metrics. It also bridges two normally separate evaluation tracks: diffusion models are often evaluated only on CRPS-style probabilistic metrics and seldom compared against strong point-forecasting baselines, whereas SimDiff is benchmarked against both.

Real-world applications (domains the paper itself cites):

• Economics and financial time series forecasting — generating future sequences from historical observations.
• Retail sales prediction — forecasting demand series where the future level may shift away from the historical level, the exact scenario Normalization Independence targets.
• Energy management — load and electricity forecasting, represented in the paper by the Electricity dataset.
• General multivariate sensor and weather forecasting — represented by the Weather and Wind datasets, where the authors report the most severe out-of-distribution drift and the largest gains from N.I.

Industry relevance. Inference cost is reported at 0.22–0.46 ms single-sample on ETTh1 versus tens to thousands of milliseconds for other diffusion models, and MoM ensembling remains competitive in total time. For production forecasting systems that must serve many series with low latency, that efficiency gap is the practical argument for a simpler architecture. The removal of a separate regressor also removes an extra model to train, version and maintain.

Future Directions

• Characterizing when Normalization Independence matters most. The ablation shows the largest gains on datasets with severe out-of-distribution drift (Weather, NorPool) and smaller gains elsewhere; the conditions under which the decoupling is essential rather than marginal remain an open question.
• Tuning the MoM ensembling parameters. The estimator depends on the number of subsamples K, subsample size B, and repetition count R; the paper notes the MoM approach has stronger concentration guarantees in finite-sample regimes, but the trade-off between K, R, number of diffusion samples and inference budget is not fully settled in the main text.
• Extending beyond the tested horizons and datasets. Evaluation uses prediction horizons of 96, 168, 192, 336 and 720 for the inference-time study and 9 multivariate datasets for MSE; whether the design holds for longer horizons, other domains, or non-multivariate settings is not established here.
• Reconciling the simplified backbone with other generative designs. The authors deliberately remove skip connections, use channel independence and drop external predictors; how much further the Transformer backbone can be simplified—or whether elements like U-Net skip paths could be reintroduced beneficially under a different noise-handling scheme—remains open.

Target Audience

Researchers and practitioners working on generative time series forecasting, especially those comparing diffusion-based models against regression-based forecasters. It will be most useful to readers already comfortable with DDPM notation and Transformer forecasting baselines, and to applied engineers who care about the accuracy-versus-inference-cost trade-off in deployed multivariate forecasting systems.

Authors’ abstract

Diffusion models have recently shown promise in time series forecasting, particularly for probabilistic predictions. However, they often fail to achieve state-of-the-art point estimation performance compared to regression-based methods. This limitation stems from difficulties in providing sufficient contextual bias to track distribution shifts and in balancing output diversity with the stability and precision required for point forecasts. Existing diffusion-based approaches mainly focus on full-distribution modeling under probabilistic frameworks, often with likelihood maximization objectives, while paying little attention to dedicated strategies for high-accuracy point estimation. Moreover, other existing point prediction diffusion methods frequently rely on pre-trained or jointly trained mature models for contextual bias, sacrificing the generative flexibility of diffusion models. To address these challenges, we propose SimDiff, a single-stage, end-to-end framework. SimDiff employs a single unified Transformer network carefully tailored to serve as both denoiser and predictor, eliminating the need for external pre-trained or jointly trained regressors. It achieves state-of-the-art point estimation performance by leveraging intrinsic output diversity and improving mean squared error accuracy through multiple inference ensembling. Key innovations, including normalization independence and the median-of-means estimator, further enhance adaptability and stability. Extensive experiments demonstrate that SimDiff significantly outperforms existing methods in time series point forecasting.