A Lightweight Sparse Interaction Network for Time Series Forecasting

AI summary

Overview

• Research area: Time series forecasting (TSF), with a focus on long-term multivariate forecasting and efficient neural architecture design. The paper sits at the intersection of linear (MLP-only) forecasting models and attention-inspired temporal interaction mechanisms.
• Technical level: Advanced. The method relies on sparse Bernoulli distributions, the Gumbel-Max trick, Gumbel-Softmax reparameterization, and a custom self-adaptive sparse regularization loss.
• Scope: The paper proposes LSINet, a lightweight MLP-only forecasting model that replaces self-attention with a learned sparse connection matrix shared across samples and variables, and validates it against advanced linear and Transformer baselines on six public datasets.

What This Paper Is About

Recent linear models for long-term time series forecasting match or beat Transformer models while being far cheaper, but they capture relationships between time steps only implicitly through stacked MLPs, which may be too weak for complex temporal dependencies. Transformers do capture these relationships explicitly through self-attention, but the attention computation is quadratic in cost and largely redundant, since the paper reports that forecasting error stays low until the attention matrix sparsity exceeds 90%. The goal of this work is to build a linear model that gets explicit temporal interaction at a fraction of the cost of self-attention, by learning a sparse, shared connection matrix over time patches.

Key Contributions

1. LSINet, a lightweight sparse interaction network for TSF. The model combines two components, a Multihead Sparse Interaction Mechanism (MSIM) and Shared Interactions Learning (SIL), and bypasses self-attention score computation entirely while still performing explicit temporal interaction. The authors state they are the first to explore a linear model equipped with efficient and explicit temporal interaction mechanisms.

2. A sparsity-induced Bernoulli connection learner. Inside MSIM, the model learns which time steps (patches) should be connected through a Bernoulli distribution over an N x N connection matrix, made differentiable via the Gumbel-Max trick and Gumbel-Softmax reparameterization, and regularized toward sparsity by a proposed self-adaptive regularization loss (ASRL).

3. Shared interaction across samples and variables. Motivated by the observation that important temporal interactions recur along the diagonal of attention heatmaps and repeat across correlated weather variables, the authors learn one shared connection matrix for all samples and variables rather than per-sample, per-variable attention, which they argue improves efficiency and convergence.

4. Extensive accuracy and efficiency validation. LSINet is compared against four advanced linear models (TimeMixer, CI-TSMixer, FiLM, DLinear) and three advanced Transformer models (PatchTST, Pathformer, Scaleformer) on Weather, Electricity, and four ETT datasets, with additional ablations on MSIM, self-attention, and ASRL, plus efficiency and heatmap visualizations.

Main Findings

• Accuracy gains over linear models (average and maximum MSE improvement ratios across all prediction lengths and datasets): CI-TSMixer 2.06% and 3.40%; FiLM 8.62% and 13.37%; DLinear 6.61% and 12.19%; TimeMixer 3.7% and 5.4%.

• Accuracy gains over Transformer models: PatchTST 2.22% and 3.55%; Scaleformer 4.73% and 6.32%; Pathformer 6.84% and 9.37%.

• Efficiency versus PatchTST and CI-TSMixer (Table 1): On Weather with batch size 64, epoch time is 59.98 s for PatchTST, 52.91 s for CI-TSMixer, and 18.92 s for LSINet; inference time is 32.73 s, 32.67 s, and 17.77 s; memory is 5034 MB, 4276 MB, and 2940 MB. On Electricity with batch size 16, epoch time is 377.25 s, 323.06 s, and 91.25 s; inference time is 200.88 s, 165.56 s, and 82.53 s; memory is 16128 MB, 13132 MB, and 6936 MB. Linear models used input length 1024 while PatchTST used 512.

• MSIM ablation: Removing MSIM increases MSE by 2.23% (average) and 6.54% (maximum) across all prediction lengths and five public datasets; replacing MSIM with self-attention increases MSE by 2.24% (average) and 6.53% (maximum).

• MSIM versus self-attention at equal input length (Table 5, input length 1024): On Weather, LSINet takes 22.36 s per epoch, 20.50 s inference, and 3026 MB memory, versus 56.45 s, 50.59 s, and 7584 MB for the self-attention variant. On Electricity, LSINet takes 91.25 s, 82.53 s, and 6936 MB versus 291.47 s, 175.13 s, and 23464 MB.

• ASRL ablation: Removing the self-adaptive sparse regularization loss increases MSE on average by 1%, 1.86%, 2.25%, and 1% for forecasting horizons 96, 192, 336, and 720 on two large datasets (Weather and Electricity).

• Sparsity case study: On a well-trained PatchTST model evaluated on the Weather test set, forecasting error remains at a similar low level until the sparsity of the attention matrix exceeds 90%.

• Repeated interaction patterns: Heatmaps of PatchTST self-attention on the Weather test set (batches of 64 test samples) show the most highlighted interactions recurring along the diagonal for a single variable, and repeating across variables such as air pressure and temperature.

• Visual similarity of learned matrices: The learned shared sparse connection matrix C at sparse rate 0.85 in LSINet produces heatmaps similar to PatchTST self-attention heatmaps with the lowest 85% of scores set to zero, both showing scattered-point and block patterns on ETTm1.

• Sample accuracy values for LSINet (Table 2, MSE ± standard deviation / MAE): Weather-96 0.147 ± 4e-4 / 0.199 ± 3e-4; Weather-720 0.304 ± 0.002 / 0.332 ± 0.002; Electricity-96 0.129 ± 1e-4 / 0.224 ± 6e-5; Electricity-720 0.192 ± 8e-4 / 0.287 ± 4e-4; ETTh1-96 0.366 ± 2e-4 / 0.391 ± 1e-6; ETTh2-96 0.267 ± 7e-5 / 0.337 ± 4e-4; ETTm1-96 0.293 ± 0.001 / 0.341 ± 8e-4; ETTm2-96 0.161 ± 5e-4 / 0.256 ± 6e-4.

Methodology in Plain English

LSINet keeps the overall shape of a linear forecaster but adds a learned interaction step between patches of the input series.

Patch encoding. The historical window is first normalized with instance normalization to reduce distribution shift between training and testing. It is then split into patches of length L with stride K (with K repeated copies of the last value appended as padding before patching). The number of patches is N = floor((n − L)/K) + 2. Patches are projected into a D-dimensional space with a trainable linear layer, and a learnable position encoding is added, giving X_d = W_p X_p + W_pos.

Learning who talks to whom. Instead of computing queries and keys, the model keeps a learnable memory embedding M for each patch. Two patch index sets (row indices and column indices of the connection matrix) are built, and their memory embeddings are concatenated, passed through a two-layer ReLU network, and turned into a probability c_ij that a connection between patch i and patch j exists. A discrete 0/1 connection z_ij is sampled from this Bernoulli distribution; because sampling is non-differentiable, the authors use the Gumbel-Max trick and then the Gumbel-Softmax reparameterization with a temperature τ so gradients flow. As τ approaches 0, the relaxed distribution becomes identical to a one-hot Bernoulli.

Forcing sparsity. Bernoulli sampling alone gives a rough, potentially noisy connection matrix. The authors intermittently extract the top K connections (K = N x N x δ) as one-hot targets and apply a cross-entropy loss between those targets and the predicted probabilities, which pushes the model to drop weak connections. δ is fixed at 0.15, meaning a sparse rate of 0.85.

Two training strategies. First, sparsity regularization is applied intermittently rather than every epoch, only when the current epoch is divisible by η (η in {1, 3}), so that sparsity exploration is not over-constrained. Second, the size of the connection matrix is capped at Ñ x Ñ by deriving patch length and stride from a fixed Ñ (set to 64) via K̃ = floor(n / Ñ) and L̃ = 2K̃, since a huge connection matrix would be hard to converge.

Sharing the pattern. Because the learned matrix is static while input windows slide over time, a time-invariant MLP mixes patch features into different temporal positions, so any input window can use the same learned connections. One shared matrix is used for all samples and all variables, rather than per-sample, per-variable attention.

Applying the connections. Multiple heads each multiply their own sparse matrix C^z_h by a projected patch feature V_h, and the results are concatenated. Temporal patch propagation is followed by a Time Updating MLP that projects the patch dimension and refreshes each patch representation. A linear alignment layer enables residual connections, and an integration MLP merges the interacted features. The output is flattened and passed to a linear predictor.

Setup. Hidden size for patch embedding, position embedding, and all MLPs is 128; the number of heads h is 4; the number of stacked STI modules is 1 on all datasets; Ñ is 64. Input length is tuned over {256, 384, 512, 768, 1024, 1536, 2048} for all compared methods, prediction lengths are 96, 192, 336, and 720, the learning rate is 1e-4, and batch size is 128 for the four ETT datasets, 64 for Weather, and 32 for Electricity. Each experiment is run 5 times with 5 different seeds for 30 epochs, on an NVIDIA GeForce RTX 3090 with PyTorch. The six datasets cover 17420 to 69680 time steps and 7 to 321 variables, and are evaluated with MSE and MAE.

Why This Matters

The paper challenges the assumption that explicit temporal interaction requires self-attention. It shows that a shared, sparse, learned connection matrix can reproduce the useful structure of attention heatmaps while being substantially cheaper, which reframes the tradeoff between the accuracy of Transformers and the efficiency of linear models. This is directly relevant to the ongoing debate over whether attention is necessary at all in long-term forecasting.

Real-world applications implied by the paper's framing and datasets:

• Weather forecasting. The paper uses the Weather dataset and motivates SIL with the example of daily temperature rising from night to noon and falling from afternoon to evening.
• Electricity load forecasting. The Electricity dataset is used, with a reported batch size of 16 in the efficiency comparison.
• Traffic flow prediction. Traffic flow is listed among the fields where TSF is crucial.
• Financial investment and medical diagnostics. Both are named as domains where TSF matters, and where lower inference memory and latency matter for deployment.

Industry relevance: the efficiency table shows linear models handling input length 1024 faster and with less memory than PatchTST handling input length 512. LSINet reports 18.92 s/epoch and 2940 MB on Weather versus 59.98 s and 5034 MB for PatchTST, and 91.25 s/epoch and 6936 MB versus 377.25 s and 16128 MB on Electricity. Lower memory and per-epoch time make long-horizon forecasting more practical on constrained hardware, and the single shared connection matrix means inference does not have to generate a matrix per sample and per variable.

Future Directions

• Relaxing the fully shared connection matrix. The authors adopt time-invariant mixing specifically because the shared matrix is static while windows slide. Whether partially personalized or step-conditioned matrices could recover still more accuracy without losing the efficiency advantage is not explored.

• Testing sparsity levels beyond the fixed 0.15 setting. δ is fixed at 0.15 (sparse rate 0.85) and Ñ at 64 with no per-dataset search. The paper does not report how accuracy varies with δ or Ñ, nor whether the reported 90% sparsity tolerance of self-attention translates to a different optimum for LSINet.

• Scaling depth. Only 1 stacked STI module is used on all datasets, and hidden size is fixed at 128 with 4 heads. Whether deeper stacks or larger hidden sizes change the accuracy and efficiency tradeoff relative to self-attention is left open.

• Broader datasets and horizons. Evaluation covers six datasets (Weather, Electricity, four ETT). The authors mention that more details will appear in a full version of the paper, and the abstract's claims rest on these public benchmarks, so generalization to other domains such as traffic, finance, and medical series named in the introduction is not directly demonstrated.

Target Audience

Researchers and practitioners working on efficient deep learning for time series forecasting, especially those comparing linear models against Transformer architectures. It is also relevant to engineers who need long-horizon multivariate forecasting under memory or latency constraints, and to readers interested in differentiable sparse structure learning, since the Bernoulli-plus-Gumbel-Softmax-plus-regularization recipe is general. Beginners will find the architecture accessible in outline, but the training objective and reparameterization details assume familiarity with attention mechanisms, sparsity regularization, and gradient estimators for discrete variables.

Authors’ abstract

Recent work shows that linear models can outperform several transformer models in long-term time-series forecasting (TSF). However, instead of explicitly performing temporal interaction through self-attention, linear models implicitly perform it based on stacked MLP structures, which may be insufficient in capturing the complex temporal dependencies and their performance still has potential for improvement. To this end, we propose a Lightweight Sparse Interaction Network (LSINet) for TSF task. Inspired by the sparsity of self-attention, we propose a Multihead Sparse Interaction Mechanism (MSIM). Different from self-attention, MSIM learns the important connections between time steps through sparsity-induced Bernoulli distribution to capture temporal dependencies for TSF. The sparsity is ensured by the proposed self-adaptive regularization loss. Moreover, we observe the shareability of temporal interactions and propose to perform Shared Interaction Learning (SIL) for MSIM to further enhance efficiency and improve convergence. LSINet is a linear model comprising only MLP structures with low overhead and equipped with explicit temporal interaction mechanisms. Extensive experiments on public datasets show that LSINet achieves both higher accuracy and better efficiency than advanced linear models and transformer models in TSF tasks. The code is available at the link https://github.com/Meteor-Stars/LSINet.