DeepBooTS: Dual-Stream Residual Boosting for Drift-Resilient Time-Series Forecasting

AI summary

Overview

Research area: Time-series (TS) forecasting, specifically robustness to concept drift — the mismatch between training and testing data distributions — analysed through a bias-variance decomposition.

Technical level: Intermediate to Advanced. The paper combines practical architecture design with formal proofs (Theorems 1–3, Appendix B), so the theory sections demand comfort with bias-variance decomposition and ensemble error decompositions.

Scope: One sentence: the paper argues that concept drift in TS forecasting is driven by prediction variance, proves that ensembling reduces that variance without increasing bias, and introduces DeepBooTS, a dual-stream residual-decreasing boosting architecture that reports an average 15.8% improvement over existing methods across multivariate benchmarks.

What This Paper Is About

Time series from natural and engineered systems are non-stationary, so the distribution a model trains on differs from the one it is tested on. This causes concept drift: validation error rises early in training even while training error keeps falling, and the paper shows this phenomenon occurs across multiple TS forecasting models and datasets. The goal is to build a forecasting architecture that directly targets the high-variance component of the error and remains accurate under distribution shift.

Key Contributions

1. A bias-variance analysis of concept drift. The authors derive an identity linking test error to prediction variance plus squared bias plus noise, and show that when mean bias and noise are fixed, the variance of the data governs the extent of concept drift. They prove (Theorem 1) that simple average ensembling gives Bias(Ȳ) = Bias(Ŷ) and Var(Ȳ) ≤ Var(Ŷ), and (Theorem 2) that a weighted ensemble has strictly lower MSE under the shifted distribution P_t, with the gap approximated as ((L−1)(1−α_l)/L)·c²σ_t² ≥ 0.

2. The DeepBooTS architecture. A dual-stream residual-decreasing boosting design in which each block of a deep model acts as an ensemble member, and block-wise outputs correct the residuals of previous blocks. The final estimate is formulated as a weighted subtraction across learners (Eq. 9), with the constraint that the odd-indexed weights minus the even-indexed weights sum to 1 and all α_l ≥ 0.

3. A variance bound for the subtraction form. Theorem 3 shows that with per-block i.i.d. errors e_l ~ N(0, ν), equal weights α, and covariance μ between blocks, Var(Ŷ) < (4/L)·α²(ν + μ). The paper states that switching the output-stream aggregation from subtraction to addition gives an approximate variance of (4/L)α²ν + 3α²μ, which is much larger.

4. Dual-stream decomposition of inputs and targets. The input stream is decomposed implicitly as X = Σ g_l(X) + R_L, while the output stream learns the residuals of the supervised signal, giving what the authors describe as a learning-driven decomposition that improves versatility and interpretability.

Main Findings

• Multivariate results: Across six datasets (ETT, Traffic, ELC, Weather, Solar, PEMS) and prediction lengths 96, 192, 336 and 720, DeepBooTS reports consistent state-of-the-art performance with an average performance increase of 15.8%. DeepBooTS averages 30 first- or second-best items across the six datasets. Two variants are configured because input lengths differ between baselines: DeepBooTS* with input length 336 and DeepBooTS with input length 96. Reported average MSE/MAE pairs include Traffic (0.373/0.261 for DeepBooTS*, 0.406/0.270 for DeepBooTS) and Weather (0.227/0.266 and 0.245/0.271).

• Univariate results: Compared with the hyperparameter-searched Periodformer, DeepBooTS yields an average 4.8% reduction across five datasets and achieves an average of 26 best terms. Under the input-96-predict-96 setting, DeepBooTS reduces MSE by 11.2% (0.143 → 0.127) on Traffic.

• Monash benchmark: On 7 Monash TS datasets (e.g., NN5, M4, Sunspot) evaluated with 7 diverse metrics (e.g., MAPE, sMAPE, MASE, Quantile) at input length 96 and output lengths {96, 192, 336, 720}, DeepBooTS achieved a score of 41 out of 54.

• Large-scale datasets: On the CBS dataset with 4,454 nodes (17GB) and the Milano dataset with 10,000 nodes (19GB), a random partitioning strategy is used because all nodes cannot fit on a single GPU. DeepBooTS reports an overall MSE reduction of 8.9% on CBS and 6.2% on Milano compared with PSLD.

• Generality: Substituting the attention mechanism changes performance considerably. Prob-Attention reduced average MSE on Electricity by 48.2% (0.311 → 0.161) and on Weather by 61.7% (0.634 → 0.243), surpassing Full-Attention. Period-Attention, Auto-Correlation and Flow-Attention also surpassed Full-Attention on those datasets.

• Component ablation: Using subtraction (−X) on the input stream beats addition (+X); on Weather the forecast error is reduced by 2.3% (0.262 → 0.256). Adding the high-speed output stream with subtraction (−Y) improves further, and adding learnable gating (G) reduces error by 4.9% on Weather (0.262 → 0.249).

• Variance and depth: DeepBooTS shows the smallest prediction variance among the compared models and the best alignment with ground truth. Serious overfitting occurs when iTransformer blocks increase from 4 to 8, while DeepBooTS deepened to 16 blocks continues to perform well.

• Interpretability: Visualising block outputs shows each block learns meaningful patterns; when the embedding dimension is low each block must learn salient patterns, and as depth increases the amplitude of shallow blocks decreases while components shift to deep blocks.

Methodology in Plain English

The authors start from the observation that normalisation methods such as RevIN make the mean of a series stationary but leave the variance unstable. Writing the error as variance + squared bias + noise, they argue that if bias and noise are held fixed, it is variance that governs how badly a model drifts. They then prove that averaging a set of predictors leaves the bias unchanged while not increasing the variance, and that under the shifted distribution the ensemble's MSE is lower than the single model's.

These results motivate a network built to behave like a boosting ensemble inside a single deep model. Each block is a fork: it takes an input X_l, transforms it into X̂_l, and passes forward the remainder R_l = X_l − X̂_l — an implicit decomposition of the input. Separately, X̂_l is projected by a predictor to the label space, producing O_l, and these outputs are subtracted from one another sequentially so that each block corrects the residual error of the previous ones. Learners can be built from fully connected, convolutional or attention layers, with an option to run attention in the frequency domain via FFT for lightweight processing; a switch δ disables the attention layer when it hurts and lets input flow straight to the feed-forward layers. A learnable gating mechanism (sigmoid applied to a linear projection) acts as the coefficient α_l controlling how fast each stream is transmitted. The result is a model whose input and output streams are both decomposed in a learning-driven way rather than by fixed preprocessing.

Why This Matters

Impact on research. The paper reframes concept drift as a variance problem and connects deep residual/boosting designs to classical ensemble theory. It offers a variance bound that suggests deep models can go deeper without escalating drift risk, and it demonstrates that the architecture can serve as a general framework into which other attention modules can be plugged.

Real-world applications (the paper explicitly states that the method has been applied to real-world scenarios, e.g., energy and power):

• Electricity consumption and power-grid forecasting, where the ELC and Electricity datasets are used.
• Traffic flow forecasting, where the Traffic and PEMS datasets are used.
• Weather and solar-energy forecasting, where the Weather and Solar datasets are used.
• Large-scale sensor networks — the CBS dataset with 4,454 nodes and Milano with 10,000 nodes model city-scale sensing infrastructure.

Industry relevance. Non-stationarity is the normal condition for deployed forecasting systems in energy, transport and environmental monitoring, and the paper reports that the method reduces error over the latest advanced baselines on 17GB and 19GB datasets that cannot fit on a single GPU, which speaks to scalability. The code is released at https://github.com/Anoise/DeepBooTS.

Future Directions

• Extending the theoretical bounds. Theorem 3 bounds variance under i.i.d. block errors with equal weights and a pairwise covariance term; relaxing these assumptions or testing the bound empirically at larger L is a natural next step. The paper notes the test error approaches Bias(f̂(X))² + σ² as L is infinite when the performance-efficiency trade-off is not considered.
• Broader attention and backbone substitution. The ablation showed large swings when the attention type was swapped, so systematically mapping which learners work best on which data regimes is an open direction.
• Scaling to larger and more heterogeneous datasets. Results are reported on CBS and Milano; whether the residual-subtraction design holds on other large-scale or foundation-model-style settings is not established in the provided content.
• Interpretability analysis in greater depth. The paper visualises block outputs and points to Appendix P for fuller analysis, but a principled account of what each block captures remains to be developed.

Note: the paper's conclusion does not state a dedicated future-work section; the items above are open questions raised by the results. Complexity analysis is referenced to Appendix G and training-time figures are not reported in the content provided.

Target Audience

Researchers and practitioners working on non-stationary time-series forecasting, especially those interested in ensembles, boosting, residual learning, and the theory of concept drift. It is also relevant to engineers deploying forecasting models in energy, transport, weather and large-scale sensor settings who need robustness under distribution shift, and to readers who want a concrete example of deriving an architecture from a bias-variance argument.

Authors’ abstract

Time-Series (TS) exhibits pronounced non-stationarity. Consequently, most forecasting methods display compromised robustness to concept drift, despite the prevalent application of instance normalization. We tackle this challenge by first analysing concept drift through a bias-variance lens and proving that weighted ensemble reduces variance without increasing bias. These insights motivate DeepBooTS, a novel end-to-end dual-stream residual-decreasing boosting method that progressively reconstructs the intrinsic signal. In our design, each block of a deep model becomes an ensemble of learners with an auxiliary output branch forming a highway to the final prediction. The block-wise outputs correct the residuals of previous blocks, leading to a learning-driven decomposition of both inputs and targets. This method enhances versatility and interpretability while substantially improving robustness to concept drift. Extensive experiments, including those on large-scale datasets, show that the proposed method outperforms existing methods by a large margin, yielding an average performance improvement of 15.8% across various datasets, establishing a new benchmark for TS forecasting.