DistDF: Time-Series Forecasting Needs Joint-Distribution Wasserstein Alignment

AI summary

Overview

Research area: Time-series forecasting, specifically the design of training objectives (learning losses) rather than neural architectures.

Technical level: Intermediate — the empirical results are easy to follow, but the core argument relies on optimal transport theory, Gaussian/Bures–Wasserstein distances, and covariance estimation.

Scope: This paper argues that the standard MSE training objective for time-series forecasting is fundamentally biased when future values are autocorrelated, and proposes a distribution-alignment loss (DistDF) that provably fixes this bias and improves accuracy across models and datasets.

What This Paper Is About

Most forecasting models are trained by minimizing mean squared error, which implicitly treats every future time step as independent. But real time series are autocorrelated — what happens at step t depends on earlier steps — so MSE is a biased estimate of the true negative log-likelihood, and this bias degrades training. DistDF instead trains the model by directly aligning the conditional distribution of the forecast sequence with that of the label sequence, using a tractable Wasserstein-based discrepancy between their joint distributions.

Key Contributions

1. Diagnosis of autocorrelation bias. The paper proves (Theorem 3.1) that the gap between MSE and the true conditional negative log-likelihood equals the difference between a covariance-weighted error norm and the ordinary Euclidean norm, vanishing only when the conditional covariance is the identity.

2. Explaining why label-decorrelation methods fall short. It shows that Fourier-based (FreDF) and PCA-based (Time-o1) label transforms only achieve marginal decorrelation, not the conditional decorrelation required to remove the bias, and empirically confirms residual correlations remain (over 50.3% of off-diagonal conditional correlations exceed 0.1 on the Traffic dataset; transforms reduce but do not eliminate this).

3. A tractable joint-distribution Wasserstein objective. DistDF minimizes the Wasserstein discrepancy between the joint distributions of (history, label) and (history, forecast). This quantity provably upper-bounds the expected conditional discrepancy of interest (Lemma 3.3), is estimable from finite data, and under a Gaussian model reduces to a closed-form Bures–Wasserstein distance over means and covariances.

4. Model-agnostic training framework. The loss is a plug-and-play regularizer combined with MSE, requiring no architectural changes, and is shown to improve a wide range of forecasting backbones.

Main Findings

• MSE is the weakest objective. In head-to-head comparisons, naive direct forecast (MSE) produced the worst MSE on ECL and Weather for the Fredformer backbone, consistent with the autocorrelation-bias theory.

• Shape-alignment losses (DILATE, Soft-DTW) give limited gains. These heuristic geometric-matching objectives partially accommodate autocorrelation but carry no unbiasedness guarantee, and their improvements over MSE were inconsistent across datasets.

• Likelihood-based losses (FreDF, Time-o1) are the strongest baselines but still biased. They substantially outperform MSE by reducing autocorrelation bias, yet they cannot eliminate it, so they never reach full conditional alignment.

• DistDF achieves the best overall results. Across TimeBridge and Fredformer backbones on ETTm1, ETTh1, ECL, and Weather (forecast lengths 96/192/336/720), DistDF attained the lowest MSE in every reported case.

• Both moments matter, and they are complementary. Ablations show that aligning only means (DistDF†) or only covariances (DistDF‡) each improves on plain DF, but combining both gives the best performance — a synergistic effect.

• The Wasserstein choice outperforms alternatives. Against EMD, MMD with linear/RBF kernels, and KL divergence, all discrepancy measures beat MSE, but the joint-distribution Wasserstein discrepancy was best in 14 out of 16 comparison cases.

• Consistent plug-and-play gains across models. Applied to various backbones, DistDF improved results throughout; on ECL it reduced MSE by up to 2.7% for iTransformer and 4.3% for Fredformer.

• Robust to the trade-off weight. The optimal γ is typically small but nonzero (roughly 0.001–0.05), showing the MSE term remains useful for preserving point-wise correspondence while the discrepancy term supplies distributional alignment. Large γ (near 1) hurts performance.

• Qualitative improvement in fine-grained tracking. Forecast plots show MSE-trained models capture overall trend but miss rapid step-to-step changes, while DistDF follows those sharper fluctuations more closely.

Methodology in Plain English

The researchers start from the observation that training a forecaster should make the distribution of its predictions match the distribution of the true future, given the same history. Directly measuring how far apart two conditional distributions are is impossible in practice, because for any given history window the dataset usually contains only one future sequence and the model outputs only one prediction — a single sample cannot describe a distribution.

Their workaround is to look at the joint distributions instead. For each batch, they concatenate the history window with the true future to form one sequence, and concatenate the same history with the model's forecast to form another. With an entire batch, these joint sequences provide plenty of samples. They then characterize each joint distribution by its mean vector and covariance matrix, and measure the mismatch using the squared 2-Wasserstein distance between two Gaussians, which has a clean closed form (mean difference plus a covariance term known as the Bures–Wasserstein distance). Theoretically, driving this joint discrepancy to zero forces the conditional distributions to match, and minimizing it also minimizes an upper bound on the conditional discrepancy you actually care about.

Because this moment-based measure ignores which prediction corresponds to which input, they do not use it alone. Instead they add it to MSE with a weight γ, so the total loss is γ times the distributional discrepancy plus (1−γ) times MSE. The MSE keeps the element-wise pairing intact; the discrepancy term supplies the distributional correction. The whole thing is differentiable and drops into standard gradient-based training without touching the model architecture.

Why This Matters

This work reframes a seemingly settled training convention — using MSE for forecasting — as a theoretically flawed choice, and gives both a proof of the flaw and a principled replacement. It shifts attention from architecture design, which dominates the literature, back to objective design, showing there is meaningful accuracy left on the table from the loss function alone. It also clarifies why recent label-transformation methods help but plateau: they target the wrong kind of decorrelation (marginal instead of conditional). Since DistDF is model-agnostic, its gains are orthogonal to and composable with architecture improvements.

Real-world applications:

• E-commerce and web traffic forecasting, where predicting next-day or next-week user traffic drives capacity planning and ad delivery; the paper's authors include researchers from Xiaohongshu (a large consumer platform).
• Robotics trajectory forecasting, where accurate short-horizon motion prediction with sharp step-to-step changes matters for safe planning.
• Manufacturing inferential sensing, where virtual sensors estimate unmeasured process variables from correlated sensor streams.
• Energy load and weather forecasting, both evaluated directly in the paper (ECL and Weather datasets) and central to grid balancing and renewable integration.

Industry relevance: the method requires no new model architecture, only a modified training loss, so it is a low-friction upgrade for teams already running Transformer or linear forecasting pipelines. The additional cost is computing means and covariance matrices per batch, which is modest relative to the model forward/backward pass. Its main practical constraint is that it needs the Gaussian/Bures–Wasserstein simplification and a tuned γ, so deployment involves a small hyperparameter search rather than a drop-in swap.

Future Directions

• Beyond second-order statistics. The Bures–Wasserstein formulation assumes Gaussianity and captures only mean and covariance; extending it to discrepancies that handle higher-order moments while staying computationally tractable is explicitly flagged as future work.

• Multivariate dependence structure. The formulation and analysis adopt a univariate setting on the grounds that autocorrelation manifests per variate, generalizing naturally to multivariate inputs — but explicitly modeling cross-variate conditional structure remains open.

• Scaling to long horizons and large batches. Covariance estimation for very long forecast lengths (T=720 is used here) and its numerical stability under limited batch sizes deserves systematic study, along with the full efficiency analysis deferred to the appendix.

• Combining with other objective families. Since shape-alignment and likelihood-based losses capture different aspects of the problem, hybrid objectives that pair Wasserstein alignment with, for example, soft-DTW or rank-based terms could be explored.

• Application to foundation-scale forecasting models. Given the trend toward pretrained time-series foundation models, testing whether distributional alignment objectives scale to pretraining (rather than per-dataset training) is a natural next step.

Target Audience

Researchers and graduate students working on time-series forecasting, particularly those focused on training objectives, loss function design, or optimal transport applications in machine learning. Practitioners at companies deploying forecasting models — in e-commerce, energy, manufacturing, or robotics — will find the plug-and-play framing and empirical gains directly actionable. Readers should be comfortable with probability distributions, covariance matrices, and Wasserstein/optimal transport concepts to fully engage with the theoretical sections; the experiments, however, are readable without that background.

Authors’ abstract

Training time-series forecasting models requires aligning the conditional distribution of model forecasts with that of the label sequence. The standard direct forecast (DF) approach resorts to minimizing the conditional negative log-likelihood, typically estimated by the mean squared error. However, this estimation proves biased when the label sequence exhibits autocorrelation. In this paper, we propose DistDF, which achieves alignment by minimizing a distributional discrepancy between the conditional distributions of forecast and label sequences. Since such conditional discrepancies are difficult to estimate from finite time-series observations, we introduce a joint-distribution Wasserstein discrepancy for time-series forecasting, which provably upper bounds the conditional discrepancy of interest. The proposed discrepancy is tractable, differentiable, and readily compatible with gradient-based optimization. Extensive experiments show that DistDF improves diverse forecasting models and achieves leading performance. Code is available at https://anonymous.4open.science/r/DistDF-F66B.