Selective Learning for Deep Time Series Forecasting

AI summary

Overview

Research area: Deep learning for time series forecasting, specifically training strategies that mitigate overfitting caused by noise and anomalies in temporal data.

Technical level: Advanced. The paper combines a practical masking strategy with information-theoretic uncertainty estimation (differential entropy of residuals), residual lower-bound estimation, and a theoretical error bound for variance estimation.

Scope: The paper proposes and evaluates a model-agnostic training strategy called selective learning, which computes the MSE loss only on a selected subset of forecast timesteps identified as generalizable, across eight real-world datasets and six deep forecasting backbones.

What This Paper Is About

Deep forecasting models are usually trained by minimizing MSE over every timestep of the forecast horizon, treating all timesteps as equally learnable. Because real-world time series contain noise (uncertain timesteps) and exogenous shocks or sensor faults (anomalous timesteps), forcing the model to fit those timesteps produces gradients that point in unreliable directions and causes overfitting. This paper asks whether the model should instead be optimized only on the timesteps that are genuinely generalizable, and it builds a mechanism to identify and mask out the rest during training.

Key Contributions

1. Selective learning as a training strategy for time series forecasting. The paper introduces a learning strategy that computes the regression loss only over a masked subset of forecast timesteps, described as the first attempt in this field to address overfitting at timestep granularity rather than at the sequence or sample level.
2. A dual-mask mechanism. Two complementary masks are combined (via an element-wise OR): an uncertainty mask based on the entropy of the prediction residual distribution, and an anomaly mask based on estimation of the residual lower bound.
3. Model-agnostic design with theoretical support. The strategy is applied without modifying backbone architectures, generates masks per variable using a channel-independent strategy, and is accompanied by a theorem bounding the variance-estimation error introduced by using residuals computed under different parameter states.
4. Extensive empirical validation. Experiments across eight real-world datasets and six baselines (Informer, Crossformer, PatchTST, iTransformer, TimesNet, TimeMixer) report improvements in all 192 evaluation cases, including in-domain, zero-shot, ablation, and training-objective comparisons. Code is released via the listed GitHub repositories.

Main Findings

• Consistent gains across every tested backbone and dataset. The paper reports that selective learning improves performance in all 192 cases measured. Example averages across prediction lengths: Informer improves from 1.289 to 0.538 MSE on ETTh1 (-58.3%) and from 1.485 to 0.494 MSE on ETTm2 (-66.8%).
• Largest benefit on overfitting-prone, early-generation models. Across datasets, the average reductions are 37.4% MSE and 25.4% MAE for Informer, and 15.6% MSE and 10.5% MAE for Crossformer.
• State-of-the-art models still improve. TimesNet shows an 8.4% MSE reduction, iTransformer 6.5% MSE and 4.4% MAE, and TimeMixer 4.3% MSE and 3.3% MAE. The paper notes these gains persist even in models already using RevIN normalization.
• Overfitting is visibly mitigated during training. On ETTh1 with prediction length F = 336, iTransformer's test MSE rises after the third epoch under standard training; selective learning yields stable convergence and an 8.1% reduction in test MSE in that setting.
• Zero-shot generalization improves. When trained on one dataset and tested on another without further training, selective learning reduced MSE by 22.6% and MAE by 14.5% on average in the challenging ETTh2 to ETTh1 and ETTm2 to ETTm1 transfer settings. On ETTh1 to ETTh2 and ETTm1 to ETTm2 transfers, the results outperformed training from scratch on the target dataset.
• Beats alternative training objectives. Using iTransformer as the backbone, selective learning achieved lower MSE/MAE than shape-based (TILDE-Q), frequency-based (FreDF), distribution-based (PS loss), and plain MSE objectives on ETTh1, ETTm2, Exchange, and Weather. On ETTh1, selective learning reached 0.415 MSE versus 0.427 for PS, 0.432 for FreDF, 0.450 for TILDE-Q, and 0.458 for MSE.
• Both masks are necessary. Ablating the uncertainty mask, the anomaly mask, or replacing the dual mask with random masking at the same ratio all degraded performance. Random masking produced results comparable to or worse than no masking, indicating the benefit comes from selecting generalizable timesteps, not from simply using fewer timesteps.
• Masking ratios depend on the data. Larger masking ratios helped on highly non-stationary datasets (ETTh1 and Exchange); smaller ratios worked better on the periodic Weather dataset. On Exchange, a 90% anomaly masking ratio gave peak performance, which the paper attributes to market-induced anomalies having greater influence than noise there.
• The choice of estimation model matters only modestly. Simpler estimation models (such as DLinear) performed better on highly non-stationary datasets like ETTh1 and Exchange, while more complex estimation models helped on the periodic Weather dataset, but the overall impact stayed limited.

Methodology in Plain English

The core idea is to hide part of the loss. At each training step, instead of averaging squared errors over all forecast timesteps, the model averages them only over the timesteps left visible by two masks, and scales by the number of visible timesteps.

The first mask targets uncertainty. Because the training set is built by sliding a window with stride 1, the same future timestep gets predicted many times within an epoch under different historical windows. The residual (prediction minus truth) at that timestep can therefore be treated as a sample of a distribution. Assuming that distribution is Gaussian, the paper computes the differential entropy of the residuals from their variance and masks the timesteps whose entropy falls in the top r_u percent, since their gradients point in essentially random directions. A theorem bounds how much error this variance estimate accumulates from the fact that the residuals were produced by slightly different parameter states, showing the error can be controlled with a small learning rate and a large batch size.

The second mask targets anomalies. Simply masking high-magnitude residuals would also throw away patterns the model has not learned yet. Instead, the paper trains a separate lightweight model to estimate the lowest residual achievable at each timestep (a residual lower bound), then computes the gap between the current residual and that bound. Truly anomalous timesteps have an elevated lower bound, so their gap is small; merely unlearned timesteps have a large gap. Timesteps whose gap falls below a threshold gamma_a (the smallest r_u percent, as written in the paper) are masked. Because the gap depends on the current predictions, the mask shifts dynamically during training rather than being fixed in advance, which the paper argues reduces distributional bias and lets the model learn the most generalizable timesteps before attempting rarer, harder ones. DLinear serves as the estimation model for all main baselines, and masks are generated independently per variable.

All baselines keep their original hyperparameters; only the masking ratios are tuned. Optimization uses Adam. Experiments use prediction lengths of 24, 36, 48, and 60 for ILI and 96, 192, 336, and 720 for other datasets, with the look-back window searched and the best result reported. Evaluation uses MSE and MAE on eight RTX 4090 24GB GPUs.

Why This Matters

Research impact. Most time series forecasting progress has come from new architectures. This paper argues the loss function itself is a leverage point, showing that masking unlearnable timesteps outperforms global sequence alignment even against newer objectives designed to match shape, frequency content, or patch-wise distributions. It also reframes overfitting in forecasting as a per-timestep phenomenon rather than a whole-model phenomenon, which gives the field a different diagnostic and intervention point.

Real-world applications:

• Traffic flow prediction, where sensor dropouts and unusual congestion events create anomalous readings.
• Weather forecasting, where rare extreme events are poorly generalizable but are mixed into training with ordinary days.
• Energy consumption planning, where usage series shift with policy and seasonality and are prone to non-stationary behavior.
• Industrial monitoring and finance, the two examples the paper gives of sensors disturbed by mechanical vibration and electromagnetic interference, and stock prices fluctuating under policy interventions.

Industry relevance. The method is a drop-in change to the training loop rather than a new architecture, requires no change to inference, and adds only a lightweight auxiliary model. It works with existing normalization techniques such as RevIN and with curriculum learning for forecasting, so teams can apply it to deployed backbones without retraining a new model family. Released code is available at the repositories linked in the paper.

Future Directions

1. Extending beyond in-domain forecasting. The paper explicitly states its scope is currently limited to in-domain time series forecasting, and suggests generalizing to classification and imputation tasks.
2. Pretraining strategies for time series foundation models. The authors list this as an intended direction, which would test whether timestep selection helps at large pretraining scale.
3. Automating the masking ratios. The paper identifies the two masking ratios as critical hyperparameters requiring guidelines (provided in appendix C.4) and shows the optimal ratio varies sharply by dataset, from small values on periodic Weather to 90% anomaly masking on Exchange. Automatically adapting those ratios per dataset remains open.
4. Reducing estimation overhead and exploring estimation model choice. The lightweight estimation model is trained alongside the backbone; whether cheaper or jointly learned estimators can match DLinear, MLP, TimeMixer, and iTransformer in this role is left open, as is the broader question of how the variance bound's requirement of a small learning rate and large batch size affects practical training cost.

Target Audience

Researchers and graduate students working on deep time series forecasting, especially those studying overfitting, noisy supervision, or training objectives rather than architectures. Practitioners with moderate deep learning backgrounds who maintain deployed forecasting models for energy, traffic, weather, or industrial telemetry will also find the method actionable, since it modifies only the training loop. Readers interested in the theoretical side will need familiarity with entropy, Lipschitz continuity, and convergence bound arguments.

Authors’ abstract

Benefiting from high capacity for capturing complex temporal patterns, deep learning (DL) has significantly advanced time series forecasting (TSF). However, deep models tend to suffer from severe overfitting due to the inherent vulnerability of time series to noise and anomalies. The prevailing DL paradigm uniformly optimizes all timesteps through the MSE loss and learns those uncertain and anomalous timesteps without difference, ultimately resulting in overfitting. To address this, we propose a novel selective learning strategy for deep TSF. Specifically, selective learning screens a subset of the whole timesteps to calculate the MSE loss in optimization, guiding the model to focus on generalizable timesteps while disregarding non-generalizable ones. Our framework introduces a dual-mask mechanism to target timesteps: (1) an uncertainty mask leveraging residual entropy to filter uncertain timesteps, and (2) an anomaly mask employing residual lower bound estimation to exclude anomalous timesteps. Extensive experiments across eight real-world datasets demonstrate that selective learning can significantly improve the predictive performance for typical state-of-the-art deep models, including 37.4% MSE reduction for Informer, 8.4% for TimesNet, and 6.5% for iTransformer.