DB2-TransF: All You Need Is Learnable Daubechies Wavelets for Time Series Forecasting

AI summary

Overview

Research area: Multivariate time series forecasting (TSF), specifically efficient Transformer alternatives for sequence modeling.

Technical level: Intermediate. The paper assumes familiarity with self-attention, wavelet decomposition, and standard forecasting evaluation protocols, though the core idea is explained with full equations.

Scope: The paper proposes DB2-TransF, an architecture that replaces the Transformer self-attention block with a learnable multi-scale Daubechies (DB2) wavelet coefficient module, and evaluates it against nine competing forecasting models across thirteen public benchmarks.

What This Paper Is About

Transformer-based forecasting models capture long-range temporal dependencies well, but self-attention has quadratic computational complexity, which limits scalability in large-scale, high-dimensional settings. The authors replace self-attention with a learnable Daubechies wavelet layer that decomposes time series into smooth (approximation) and noise/detail components at multiple scales, aiming to retain predictive accuracy while reducing memory and compute cost. The goal is a scalable, resource-efficient forecasting framework that works in resource-constrained environments.

Key Contributions

1. DB2-TransF architecture: A forecasting model, described as linear in complexity, that substitutes the standard self-attention mechanism with a novel learnable Daubechies wavelet module, targeting both predictive accuracy and computational efficiency.
2. Learnable multi-scale DB2 design: The low-pass and high-pass DB2 filter coefficients (α^k, β^k) are made trainable, initialized close to their classical values, and applied recursively across L decomposition levels.
3. Reported performance gains: The model is reported to show consistent gains over leading forecasting methods while reducing computational requirements compared with earlier Transformer and MLP-based models.
4. Broad empirical study: Detailed experiments covering 13 diverse time series forecasting datasets, presented as evidence of effectiveness and generalization.

Main Findings

• Accuracy is competitive with, but not uniformly better than, the strongest baselines. Across the dataset average rows reported in the tables, DB2-TransF achieves the lowest average MSE among the ten compared models on Electricity (0.176 MSE / 0.270 MAE, versus iTransformer at 0.178 / 0.270) and the lowest average error on Exchange (0.335 / 0.391 versus iTransformer at 0.360 / 0.403).

• Other reported average wins over iTransformer: Weather (DB2-TransF 0.252 / 0.278 versus 0.258 / 0.278), PEMS04 (0.108 / 0.221 versus 0.111 / 0.221), PEMS07 (0.094 / 0.203 versus 0.101 / 0.204), ETTm1 (0.394 / 0.403 versus 0.407 / 0.410), ETTm2 (0.287 / 0.331 versus 0.288 / 0.332), and ETTh2 (0.370 / 0.400 versus 0.383 / 0.407).

• Cases where iTransformer is better: Traffic (DB2-TransF 0.436 / 0.287 versus iTransformer 0.428 / 0.282), Solar-Energy (0.248 / 0.279 versus 0.233 / 0.262), PEMS03 (0.134 / 0.240 versus 0.113 / 0.221), and PEMS08 (0.155 / 0.241 versus 0.150 / 0.226).

• Strong long-horizon behavior on Electricity: At the 720-step horizon, DB2-TransF reports 0.212 MSE / 0.307 MAE versus iTransformer at 0.225 / 0.317.

• Memory reduction is claimed but not quantified in the available text. The abstract asserts "substantially reducing memory usage," but no memory, parameter-count, or runtime figures appear in the content provided.

• Baseline set: Nine models are compared — iTransformer, PatchTST, Crossformer, FEDformer, Autoformer (Transformer-based), RLinear, TiDE, DLinear (linear-based), and TimesNet (temporal convolution-based).

• Evaluation protocol: Results are reported as average MSE and MAE across forecast horizons of 96, 192, 336, and 720 for Electricity, Exchange, Weather, Solar-Energy, Traffic, and the ETT datasets, and horizons of 12, 24, 48, and 96 for the PEMS datasets, following the protocol used in the iTransformer work.

• Caveat on reported numbers: Several entries in the results tables appear internally inconsistent in the supplied text — for example, the ETTh1 average row lists values such as 0.254 in the iTransformer MSE and MAE columns and 0.254 in the PatchTST MAE column, and some individual cells carry MAE values that do not align with their MSE values. The ETTh2 average row is cut off. The ETTh1 averages and any conclusions drawn from them should be treated with caution until the full tables are verified.

• Training setup: AdamW optimizer, exponential decay learning-rate scheduler, early stopping patience of 5 epochs, Python 3.10.16, CUDA 12.1, Ubuntu 24.04.1 LTS, and a single RTX 4060 GPU.

• Datasets used: Traffic (862 variates, 17,544 timesteps, 1-hour granularity), PEMS03/04/07/08 (358/307/883/170 variates, 26,209/16,992/28,224/17,856 timesteps, 5-minute granularity), ETTm1/ETTm2/ETTh1/ETTh2 (7 variates, 17,420/69,680 timesteps, 15-minute / 1-hour granularity), Electricity (321 variates, 26,304 timesteps, 1 hour), Exchange (8 variates, 7,588 timesteps, 1 day), Weather (21 variates, 52,696 timesteps, 10 minutes), and Solar-Energy (137 variates, 52,560 timesteps, 10 minutes).

• Ablation study results are not reported in the provided content, despite the results section stating that an ablation study is presented.

Methodology in Plain English

The classical building block. The standard DB2 wavelet transform uses four fixed low-pass filter coefficients (h_f) and four fixed high-pass filter coefficients (g_f), defined by expressions involving the square root of 3. Sliding these filters over a signal of even length n produces approximation coefficients A[n], which capture slow, smooth trends, and detail coefficients D[n], which capture fast variation and noise.

Making it learnable. The authors replace the fixed filters with trainable d-dimensional vectors α^k and β^k, initialized approximately at the classical DB2 values and then optimized by backpropagation. The approximation and detail coefficients are computed as element-wise weighted sums of the four learnable filter taps over a sliding window of the input.

Going multi-scale. Level 0 produces A and D from the raw input. At each subsequent level l, the approximation output of the previous level becomes the new input (x_{l+1} = A_l), producing a sequence of detail coefficient sets [D_1, …, D_L] plus a final approximation A_L. Together these form a multi-resolution representation of the original series.

The DB2-TransF block (MLDB). The input is layer-normalized and split across H heads. Each head runs an L-level learnable DB2 decomposition, with zero padding and window unfolding (window size and stride typically chosen as multiples of 2, since the filters have four even-indexed coefficients). Each head's final output concatenates the last approximation with all detail coefficient sets. Head outputs are concatenated along the feature axis, projected back to the input dimension by a linear layer, temporally trimmed to align with the input, and combined with a residual connection. A standard feedforward network with two MLP layers and GELU activation follows, wrapped in its own residual connection. This whole structure is repeated N times, and a forecasting head maps the encoded sequence to the output of shape B × T_pred × D.

What changed versus a Transformer. The query-key-value projection and softmax attention are removed entirely. Wavelet decomposition supplies the multi-scale, cross-variable mixing instead. Layer normalization, residual connections, multi-head structure, and the feedforward network are retained.

Why This Matters

Impact on research. The paper adds to a line of work questioning whether attention is necessary for time series forecasting, and offers a signal-processing-grounded alternative: a fixed-form wavelet decomposition whose coefficients are learned end-to-end. It connects classical multiresolution analysis to modern sequence architectures and provides a benchmark set spanning traffic, energy, weather, and exchange-rate domains for testing such replacements.

Real-world applications:

• Traffic flow forecasting, using the Traffic dataset (862 sensors in the San Francisco Bay Area) and the PEMS datasets from California highway systems, where sensor readings are sampled every 5 minutes and much of the data is redundant.
• Electricity load forecasting for 321 customers, where strong daily and weekly periodicity suits multi-scale decomposition.
• Solar power generation forecasting across 137 PV plants, where smooth irradiance trends and high-frequency disturbances coexist.
• Weather forecasting from 21 meteorological features sampled every 10 minutes, including sparse and largely aperiodic signals.
• Financial exchange-rate forecasting for 8 countries' daily rates (1990–2016), a low-variate, mostly aperiodic setting.

Industry relevance. The authors explicitly motivate the work by deployment in resource-constrained environments, and all experiments run on a single RTX 4060 GPU. That matters for edge and on-premises forecasting deployments where multi-GPU Transformer training is not feasible. Sensor networks, utilities, and transportation operators continuously sample at high rates and generate largely redundant readings, exactly the setting where a decomposition that separates smooth trends from detail is relevant.

Future Directions

• Quantify the efficiency claim. The paper repeatedly claims reduced memory usage and linear complexity, but no memory, FLOP, runtime, or parameter-count measurement appears in the available content. Reporting these against iTransformer and PatchTST at matched horizons is the most direct open task.
• Publish the ablation study. The results section announces an ablation study, but none of its results appear in the provided text. Isolating the effect of the learnable coefficients versus fixed DB2 coefficients, the number of levels L, and the number of heads H would clarify which components drive performance.
• Investigate the losses. DB2-TransF trails iTransformer on Solar-Energy, Traffic, PEMS03, and PEMS08, with the Solar-Energy gap being the largest of these. Understanding whether these datasets are dominated by high-frequency structure that the detail coefficients under-represent would guide architectural adjustment.
• Explore the wavelet design space. Only the four-coefficient DB2 wavelet is used. Other Daubechies orders, other wavelet families, and adaptive choices of decomposition level or window size are natural extensions.
• Extend beyond forecasting. The paper's review notes that models such as TimesNet handle forecasting, classification, and anomaly detection jointly, and one figure caption refers to "time series classification," suggesting the architecture may transfer to other time series tasks.

Target Audience

This paper suits graduate students and researchers working on efficient sequence models or wavelet-based deep learning, and practitioners who need forecasting on limited hardware. Readers should be comfortable with attention mechanics, discrete wavelet transforms, and standard MSE/MAE forecasting benchmarks. Those specifically interested in attention-free architectures, frequency-domain inductive biases, or multivariate forecasting across heterogeneous domains (traffic, energy, weather, finance) will find it most relevant. Readers looking for verified memory-savings numbers or a completed ablation will need to consult the full paper and the linked code repository at https://github.com/SteadySurfdom/DB2-TransF.

Authors’ abstract

Time series forecasting requires models that can efficiently capture complex temporal dependencies, especially in large-scale and high-dimensional settings. While Transformer-based architectures excel at modeling long-range dependencies, their quadratic computational complexity poses limitations on scalability and adaptability. To overcome these challenges, we introduce DB2-TransF, a novel Transformer-inspired architecture that replaces the self-attention mechanism with a learnable Daubechies wavelet coefficient layer. This wavelet-based module efficiently captures multi-scale local and global patterns and enhances the modeling of correlations across multiple time series for the time series forecasting task. Extensive experiments on 13 standard forecasting benchmarks demonstrate that DB2-TransF achieves comparable or superior predictive accuracy to conventional Transformers, while substantially reducing memory usage for the time series forecasting task. The obtained experimental results position DB2-TransF as a scalable and resource-efficient framework for advanced time series forecasting. Our code is available at https://github.com/SteadySurfdom/DB2-TransF