Naga: Vedic Encoding for Deep State Space Models

AI summary

Overview

Research area: Long-term time series forecasting (LTSF) with deep State Space Models (SSMs), specifically a Vedic-mathematics-inspired encoding applied to a Mamba2-style cell.

Technical level: Intermediate — the core idea (reverse the sequence, project both directions, multiply element-wise) is simple, but the surrounding architecture is a deep SSM with state-space recurrences, SSD matrices, and gated convolution blocks.

Scope (one sentence): The paper proposes and evaluates Naga, a deep SSM architecture whose bidirectional "Vedic" element-wise encoding is benchmarked against 28 existing models on seven standard LTSF datasets across prediction horizons of 96 to 720.

What This Paper Is About

Long-range forecasting models must capture dependencies between time steps that are far apart, and existing deep SSMs such as Mamba2 do this with dense matrix multiplications that entangle the model's internal representations. The authors borrow the "vertically and crosswise" decomposition idea from Vedic mathematics and turn it into an architectural layer: the input sequence and its time-reversed copy are projected separately and then combined by element-wise (Hadamard) multiplication, producing a "Vedic encoding" before the sequence enters a Mamba2 cell. The goal is to improve forecasting accuracy while keeping the model small and computationally efficient.

Key Contributions

1. The Naga model — a deep SSM architecture that integrates Vedic mathematical principles into the existing Mamba2 framework by replacing dense multiplicative interaction with a bidirectional element-wise decomposition.
2. Experimental evidence that the Vedic decomposition of multiplications helps deep SSMs learn long time-series forecasting tasks, validated through internal and external ablation studies.
3. A mathematical argument that the Vedic decomposition introduces an inductive bias which enhances gradient flow and representation disentanglement within deep state space models.
4. Benchmark results across seven commonly used LTSF datasets and multiple sequence lengths, in which Naga outperforms state-of-the-art methods in MSE.

Main Findings

• Benchmark performance across seven datasets: On ETTh1 (horizon 96) Naga reports MSE 0.124 after 16 epochs (131.02 s runtime); ETTh2 (720) MSE 0.139, 73 epochs, 213.31 s; ETTm1 (720) MSE 0.117, 35 epochs, 2145.10 s; ETTm2 (96) MSE 0.119, 23 epochs, 275.82 s; Weather (96) MSE 0.089, 45 epochs, 812.64 s; Traffic (96) MSE 0.006, 38 epochs, 1150.77 s; ILI (24) MSE 1.606, 81 epochs, 3802.77 s.
• Outperforms 28 state-of-the-art models: The abstract states Naga outperforms 28 current state-of-the-art models and is more efficient than existing deep SSM-based approaches. Prediction horizons evaluated run from 96 to 720.
• Small parameter footprint: Naga has only 2.1 million parameters, yet matches or surpasses much larger transformer-based architectures such as Autoformer, Informer, and xPatch, which the paper says typically require six to nine times more parameters.
• Internal ablation on ETTh1 (Table 3): The best configuration reaches RMSE 0.1078 and MAE 0.0776. Removing the Vedic encoding degrades RMSE by -4.78% and MAE by -3.57%. Removing the Naga cell degrades RMSE by -19.28% and MAE by -26.39%. Raising the input masking probability to 0.1 degrades RMSE by -8.39% and MAE by -10.95%. The configuration listed as "Vedic, Naga, Flip, Mask=0.0" gives RMSE 0.1085 and MAE 0.0783 (-0.69% / -0.80%).
• External ablation on ETTh1 (Table 5): Adding Inter-Channel Attention (E3) produced MSE 0.07 / MAE 0.06 at 70 s runtime, and Dynamic Kernel Parametrization (E4) produced MSE 0.07 / MAE 0.05 at 140 s runtime — the lowest errors and the longest runtime among modified architectures. Removing LayerNorm (E1, 50 s) gave MSE 0.16 / MAE 0.27; Gated Activation (E2, 50 s) gave 0.17 / 0.27; SSD-Structure Integration (E5, 130 s) gave 0.17 / 0.26; Feedforward Reconstruction Head (E6, 170 s) gave 0.17 / 0.27; Removing Bidirectionality (E7, 160 s) gave 0.16 / 0.26.
• Original baselines under identical conditions: SMamba scored MSE 0.32 / MAE 0.41 at 160 s; LTSMamba scored 0.29 / 0.39 at 130 s; Mamba2 scored 0.30 / 0.40 at 130 s. SMamba shows the highest error values of all compared configurations.

Methodology in Plain English

The input sequence is first flipped along the time axis. Two separate linear projections are applied — one to the original sequence, one to the reversed sequence — and the two results are multiplied together element by element, then passed through dropout. This is the Vedic encoding, and its purpose is to pair each time index t with its symmetric counterpart T − t + 1, creating a bilinear interaction that links distant time steps.

The encoded representation is then fed into a Mamba2-style cell. A linear projection expands it into three parts: a z component of size d_inner, an x_bc component of size d_inner, and a dt component of size 2·d_state + h_head. The x_bc part passes through a 1D convolution followed by a SiLU activation, then is normalized along the feature dimension using the mean and variance. The first d_inner/2 feature channels of this normalized output form the Mamba2 hidden state. A single linear head maps the last time step's hidden state to the full predicted sequence of length pred_len.

Training minimizes MSE against ground truth. Experiments used early stopping with patience of five epochs and a minimum delta of 10⁻⁴, a maximum of 100 epochs (never reached in any run), no learning-rate scheduler or warm-up, and no pretrained weights. Every run was repeated 10 times. Hardware was a Tesla P100-PCIE-16GB GPU (17.1 GB memory, 4 MB L2 cache, warp size 32) on Linux 6.6.56+ (x86_64) with glibc 2.35, using the Adam optimizer at learning rate 0.003581, weight decay 0.0001, batch size 64, and random seed 42. Deterministic cuDNN settings were enforced for reproducibility.

Datasets: Weather (21 meteorological variables at 10-minute resolution, ~1 year, 70/15/15 split), Traffic (hourly occupancy from 862 California freeway sensors, 33,792 training / 4,096 validation / 4,096 test samples), ILI (weekly CDC influenza-like illness statistics, 70/15/15 split), and the four ETT variants (seven variables each; ETTh1/ETTh2 hourly with 60/20/20 splits; ETTm1 and ETTm2 at 15-minute sampling with 20,160 training and 11,520 validation and test samples each). Normalization statistics were computed on the training subset only.

Why This Matters

Impact on research. The paper argues that a deliberately structured inductive bias — component-wise temporal interaction rather than fully entangled matrix products — can improve generalization under limited or noisy data and make learned state transitions more transparent. If the reported results hold, this suggests that architectural borrowing from symbolic or ancient mathematical decompositions is a viable route to more interpretable deep sequence models, and that competitive LTSF accuracy does not require large parameter counts.

Real-world applications (domains the paper names or evaluates):

• Finance — long-range temporal dependencies in financial series.
• Energy systems — electricity transformer temperature and load forecasting (the ETT benchmarks).
• Climate and meteorology — the Weather dataset's temperature, humidity, pressure, and wind-speed variables.
• Transportation — traffic occupancy prediction from freeway sensor networks.
• Public health — weekly influenza-like illness surveillance via the ILI dataset.

Industry relevance. The 2.1 million parameter footprint and the moderate runtimes reported (e.g., 131.02 s for ETTh1 at horizon 96) point toward deployment scenarios where memory and compute budgets are constrained. The ablation table shows an explicit accuracy-versus-runtime trade-off: simple modifications (E1, E2) run in 50 s but are less precise, while E3 (70 s) and E4 (140 s) achieve the lowest errors — giving practitioners a menu of configurations rather than a single fixed model.

Future Directions

• Resolution of internal inconsistencies. The paper's prose labels in the internal ablation (attributing -4.78%/-3.57% to "I1" and -19.28%/-26.39% to "I2") do not match the row labels in Table 3, where those values appear under I2 ("Without Vedic") and I3 ("No Naga") respectively, and where the best RMSE 0.1078 is listed under "No Flip" while the text calls flip augmentation part of the baseline. Clarifying which component each number belongs to would be a natural next step.
• Trade-off between accuracy and runtime. E3 and E4 achieve the lowest MSE and MAE but cost 70 s and 140 s respectively, versus 50 s for E1 and E2. Whether Inter-Channel Attention and Dynamic Kernel Parametrization can be combined or made cheaper without losing their accuracy benefit remains open.
• Extension beyond the evaluated settings. The paper reports on seven benchmarks with horizons between 96 and 720; generalization to other sampling rates, other domains, and longer horizons is not established.
• Deeper validation of the mathematical claim. The paper asserts that the Vedic decomposition enhances gradient flow and representation disentanglement, but the ablation evidence presented is performance-based; direct measurement of gradient behavior or representation structure is not reported.

Target Audience

The primary audience is machine learning researchers working on time series forecasting and state space models, particularly those already familiar with Mamba and Mamba2 architectures. It also suits practitioners who need compact, efficient forecasting models and who are willing to read architecture-level detail to choose among the ablated configurations. Readers primarily interested in Vedic mathematics itself will find it used as an inspiration for a structural design pattern rather than as a formal framework.

Authors’ abstract

This paper presents Naga, a deep State Space Model (SSM) encoding approach inspired by structural concepts from Vedic mathematics. The proposed method introduces a bidirectional representation for time series by jointly processing forward and time-reversed input sequences. These representations are then combined through an element-wise (Hadamard) interaction, resulting in a Vedic-inspired encoding that enhances the model's ability to capture temporal dependencies across distant time steps. We evaluate Naga on multiple long-term time series forecasting (LTSF) benchmarks, including ETTh1, ETTh2, ETTm1, ETTm2, Weather, Traffic, and ILI. The experimental results show that Naga outperforms 28 current state of the art models and demonstrates improved efficiency compared to existing deep SSM-based approaches. The findings suggest that incorporating structured, Vedic-inspired decomposition can provide an interpretable and computationally efficient alternative for long-range sequence modeling.