Quadratic Direct Forecast for Training Multi-Step Time-Series Forecast Models

AI summary

Overview

Research area: Machine learning / deep learning for time-series forecasting, specifically the design of training objectives (loss functions) for multi-step forecasting models.

Technical level: Intermediate. The paper assumes familiarity with mean squared error, covariance matrices, negative log-likelihood, and gradient-based optimization, but its central idea can be understood without deep mathematical background. Some parts (bilevel optimization, Cholesky reparameterization, meta-learning comparisons) require more advanced knowledge.

Scope: The paper proposes a new training objective and learning algorithm—Quadratic Direct Forecast (QDF)—that improves multi-step time-series forecasting by explicitly modeling correlations between future steps and assigning different weights to different forecasting horizons.

What This Paper Is About

Most time-series forecasting models are trained with mean squared error (MSE), which treats every future time step as an independent, equally important prediction task. The authors show this is flawed in two ways: future steps are actually correlated with one another even after conditioning on the observed history, and different steps have different difficulty and uncertainty, so they deserve different weights. The paper derives a weighted quadratic objective from likelihood theory, then proposes QDF—an algorithm that learns the weighting matrix from data and uses it to train any existing forecasting model.

Key Contributions

1. Identification of two concrete problems in forecasting objectives. The authors formalize the label autocorrelation effect (correlated future steps) and heterogeneous task weights (unequal importance of different horizons) as the two obstacles that standard MSE fails to address, supported by a case study on the ECL dataset.

2. A quadratic-form weighted training objective. They derive an objective of the form (Y − Ŷ)ᵀ Σ⁻¹ (Y − Ŷ), where the off-diagonal entries of the weighting matrix capture autocorrelation between future steps and the non-uniform diagonal entries encode per-step task weights.

3. The QDF learning algorithm. Because the true conditional covariance is unknown and cannot be estimated from a single label sequence, QDF treats the weighting matrix as learnable parameters and optimizes it via a bilevel procedure: an inner loop trains the forecast model on one data split, and an outer loop updates the weighting matrix on a disjoint holdout split to target generalization. A Cholesky reparameterization keeps the matrix positive semi-definite.

4. Broad empirical validation. QDF is evaluated across eight datasets, ten baseline models, five alternative objectives, four meta-learning optimizers, and extensive ablations, consistently achieving state-of-the-art results.

Main Findings

• QDF improves accuracy across all tested datasets. Using TQNet as the backbone, QDF achieves the best MSE and MAE on ETTm1/ETTm2/ETTh1/ETTh2, ECL, Weather, PEMS03, and PEMS08. On PEMS08 it lowers both MSE and MAE by 0.019 relative to the strongest baseline.

• Both ingredients of the weighting matrix matter, and they combine. The ablation shows QDF† (only heterogeneous diagonal weights) and QDF‡ (only off-diagonal autocorrelation modeling) each beat plain MSE-based direct forecasting, while full QDF outperforms both, indicating a synergistic effect.

• QDF beats alternative objectives. Compared with Time-o1, FreDF, Koopman, Soft-DTW, and standard direct forecast loss on both TQNet and PDF, QDF delivers the best average results. The authors argue Time-o1 and FreDF only achieve marginal decorrelation and still weight components equally.

• The objective is model-agnostic. Applying QDF to TQNet, PDF, Fredformer, and iTransformer yields consistent gains; on ECL, FredFormer's MSE drops 7.4% and TQNet's drops 5.9%.

• Established meta-learning optimizers can also learn the weighting matrix, but QDF's approach is stronger. MAML, iMAML, MAML++, and Reptile all improve over the identity-matrix baseline, yet QDF wins because those methods do not explicitly optimize for out-of-sample generalization (on ECL at T=720, QDF reduces MSE by 7.37%).

• Hyperparameters are forgiving. Performance improves sharply with the number of inner updates, is best at K=3 data splits, and remains robust across a wide range of the update rate η; setting η to zero recovers the direct-forecast baseline.

• A case study confirms the motivating problem is real. On ECL with a 96-step horizon, over 61.4% of off-diagonal partial correlations exceed 0.1, and conditional variances vary substantially across future steps—while the transformed components of FreDF and Time-o1 retain nonzero residual correlations.

Methodology in Plain English

The authors start from a statistical fact: if forecast errors follow a multivariate Gaussian distribution, the correct training loss is a quadratic form weighted by the inverse of the conditional covariance matrix of the future values. Under this lens, standard MSE is just a special case that assumes the covariance is the identity matrix—i.e., that future steps are uncorrelated and equally noisy. Neither assumption holds for real time series.

Estimating that covariance directly is impractical, because for each historical window there is only one observed future sequence, so the authors cannot simply compute an empirical covariance. Instead, they flip the problem: treat the weighting matrix itself as a set of learnable parameters, and choose it so that the model trained with it generalizes well.

This creates a two-level optimization. An inner loop trains the forecast model on one portion of the training data using the current weighting matrix. An outer loop then measures the loss on a held-out portion and updates the weighting matrix—crucially, the gradient flows through the model parameters, so the update reflects how changing the weights changes the model's generalization behavior rather than just fitting the holdout. To keep the matrix valid (symmetric positive semi-definite), it is factorized as LLᵀ with a lower-triangular L whose diagonal is passed through a softplus, turning the constrained problem into an unconstrained one solvable with standard gradient descent.

The full QDF workflow initializes the weighting matrix to the identity, splits the training set chronologically into K subsets for robustness against overfitting to any single regime, repeatedly runs the atomic update across subsets until the matrix converges (measured by Frobenius norm change below 1e-4), and finally trains the forecasting model from scratch with the learned objective. No test data is used at any point, so there is no leakage, and the procedure is independent of the model architecture.

Why This Matters

Impact on research: The paper shifts attention from architecture design—where most forecasting research has concentrated—back to the training objective, showing that a better loss can improve state-of-the-art models without changing a single layer. It connects forecasting to likelihood theory and meta-learning, and offers a reusable, model-agnostic recipe that others can apply to new architectures. The distinction between marginal and conditional decorrelation clarifies why earlier decorrelation-based objectives only partially fix the problem.

Real-world applications:

• Energy load forecasting: Utilities use multi-step forecasts of electricity consumption (the ECL dataset) for grid scheduling; better handling of correlated and unevenly uncertain demand steps can reduce reserve requirements.
• Weather and climate prediction: Forecast horizons differ greatly in predictability, and heterogeneous weighting fits that structure naturally.
• Financial and stock forecasting: Returns across future days are correlated and vary in volatility, making a fixed equal-weight loss a poor fit.
• Traffic and transportation planning: PEMS-style sensor forecasting for congestion management benefits from accurate long-horizon prediction where later steps are harder.

Industry relevance: The method is a drop-in training-time enhancement requiring no inference-time cost, which matters for latency-sensitive production systems. It was developed with industrial co-authors (Xiaohongshu Inc.) and evaluated on large-scale public benchmarks, suggesting direct applicability to demand forecasting, recommendation-adjacent prediction, and operational analytics pipelines where retraining with a better loss is cheaper than redesigning a model.

Future Directions

• Extending QDF beyond forecasting. The authors note that the same issues—correlated targets and unequal task difficulty—appear in user rating prediction and dense image prediction, making those natural extensions.
• Replacing the static weighting matrix with a hypernetwork. The current objective is fixed once learned; a hypernetwork that generates the objective dynamically could be more expressive and potentially better performing.
• Understanding when autocorrelation modeling pays off most. Whether the learned matrix differs systematically across dataset domains, frequencies, and horizon lengths is an open empirical question the paper does not fully explore.
• Scaling and stability of the bilevel procedure. Runtime and convergence behavior under very long horizons, very large datasets, or non-stationary distributions (where a fixed objective may go stale) remain open practical questions.

Target Audience

Machine learning researchers working on time-series forecasting, especially those interested in loss function design, likelihood-based objectives, or meta-learning. Practitioners who train forecasting models in industry (energy, finance, weather, traffic) will find the method directly usable since it plugs into existing architectures. Graduate students with a background in statistics or optimization will benefit from the derivation linking conditional covariance to training objectives. Readers looking for a new architecture will not find one here—this paper is for those who want to improve the training signal instead.

Authors’ abstract

The design of training objective is central to training time-series forecasting models. Existing training objectives such as mean squared error mostly treat each future step as an independent, equally weighted task, which we found leading to the following two issues: (1) overlook the label autocorrelation effect among future steps, leading to biased training objective; (2) fail to set heterogeneous task weights for different forecasting tasks corresponding to varying future steps, limiting the forecasting performance. To fill this gap, we propose a novel quadratic-form weighted training objective, addressing both of the issues simultaneously. Specifically, the off-diagonal elements of the weighting matrix account for the label autocorrelation effect, whereas the non-uniform diagonals are expected to match the most preferable weights of the forecasting tasks with varying future steps. To achieve this, we propose a Quadratic Direct Forecast (QDF) learning algorithm, which trains the forecast model using the adaptively updated quadratic-form weighting matrix. Experiments show that our QDF effectively improves performance of various forecast models, achieving state-of-the-art results. Code is available at https://anonymous.4open.science/r/QDF-8937.