TEFormer: Structured Bidirectional Temporal Enhancement Modeling in Spiking Transformers

AI summary

Overview

• Research area: Neuromorphic computing and brain-inspired machine learning, specifically Spiking Neural Networks (SNNs) and Spiking Transformers for temporal sequence modeling.
• Technical level: Advanced. The paper assumes familiarity with LIF neurons, spiking self-attention, Transformer architecture, and neuron encoding schemes (rate, phase, TTFS, direct).
• Scope: This paper proposes TEFormer, a Spiking Transformer that adds forward and backward temporal fusion to the attention and MLP components, and benchmarks it across static, neuromorphic, and temporally complex datasets under a unified STEP/BrainCog evaluation platform.

What This Paper Is About

Spiking Transformers can model sequences, but existing methods mostly move temporal information in one direction (past to present), often using heuristic, dataset-sensitive hyperparameters or step-wise sequential operations that break the parallelism of attention. TEFormer addresses this by decoupling temporal modeling across the two core Transformer components, adding a fully parallel forward temporal fusion in attention and a backward gated recurrence in the MLP, inspired by feedforward–feedback modulation in the human visual pathway.

Key Contributions

1. Temporal Enhanced Attention (TEA): A parallel forward temporal fusion mechanism for spiking attention that uses only a single learnable scalar and no manually tuned hyperparameters, preserving Transformer parallelism.
2. Temporal MLP (T-MLP): A restructured MLP with a lightweight single-gate backward recurrence that aggregates information in reverse time order, replacing the conventional up-sampling pathway.
3. Bidirectional temporal fusion framework: Combining TEA (forward) and T-MLP (backward) into a unified bio-inspired design that the authors describe as the first such framework in Spiking Transformers.
4. First systematic encoding-scheme evaluation for Spiking Transformers: Benchmarking under direct, phase, rate, and TTFS encodings to test whether temporal gains hold across different spiking representations.

Main Findings

• CIFAR10/CIFAR100 accuracy: TEFormer reaches 96.24% on CIFAR10 and 79.84% on CIFAR100, the only model in Table 1 above 96% on CIFAR10. Baselines include QKFormer (95.91 / 79.09), SDT (95.78 / 78.64), Spikformer (95.09 / 77.72), and TIM (94.20 / 75.04) under the same STEP framework.
• Neuromorphic datasets: TEFormer achieves 81.90% on CIFAR10-DVS (vs. Spikformer 80.25, QKFormer 79.30, TIM 80.85), 78.50% on N-CALTECH101 at 2-256 size, 78.05% at 4-384 size, and 95.95% on NCARS.
• Temporally complex datasets: 80.94% on sCIFAR, 96.20% on sMNIST, 90.19% on SHD, 63.65% on HMDB51-DVS, and 63.16% on UCF101-DVS, outperforming QKFormer and TIM across these tasks.
• Robustness across encodings: Under Step=4 on CIFAR10, TEFormer scores 96.24 (Direct), 89.92 (Phase), 84.74 (Rate), and 87.46 (TTFS), leading all baselines in every encoding.
• Both modules are necessary: Baseline alone scores 95.91; adding T-MLP alone gives 95.98; adding TEA alone gives 95.85 (a degradation); combining both reaches 96.24.
• Direction matters: The optimal configuration pairs TEA forward (→) with T-MLP backward (←), scoring 96.24, versus 96.06 (←/→), 96.09 (←/←), and 95.95 (→/→).
• Single-gate design wins: TEFormer (7.77M params) scores 96.24 / 79.84 / 96.88 on CIFAR10/CIFAR100/SVHN, beating the GRU-style variant (95.90 / 79.15 / 96.79, 9.43M params) and the EMA-style variant (96.05 / 79.55 / 96.72, 6.85M params).
• Learnable alpha helps: On CIFAR10, learnable alpha gives 96.24 versus 96.16 (fixed 0.5) and 96.04 (fixed 0.75); learned values were 0.66/0.59 across the two stages. On CIFAR10-DVS, learned 0.5/0.5 matches fixed 0.5 at 81.9.
• Efficiency trade-off: At MLP ratio 4, TEFormer has 1.736G MLP FLOPs and 5.18 mJ estimated total energy; at MLP ratio 2, it has 0.642G FLOPs and 2.39 mJ, which is below Spikformer (3.32 mJ), TIM (3.32 mJ), and above SDT (1.09 mJ), while still scoring 95.98%.
• Two enhanced layers is optimal: Applying temporal enhancement to 1, 2, 3, or 4 layers yields 95.24, 96.04, 95.82, 95.66 on CIFAR10, with parameter counts of 6.96M, 7.77M, 11.01M, and 14.25M respectively.

Methodology in Plain English

The authors build on the STEP evaluation platform, which standardizes training and evaluation for Spiking Transformers on top of the BrainCog framework so that different models see identical data loading, augmentation, and training settings.

For TEA, the model computes weights over all previous time steps using an exponential moving average controlled by a single scalar. That scalar is passed through a sigmoid to produce a decay factor between 0.5 and 1, and the resulting weights form a mask matrix applied to the value tensor in one matrix multiplication. Because this is a single matrix multiply rather than a loop over time steps, all time steps are fused in parallel. TEA adds no convolutional or linear layers and is applied only on the value pathway in combination with QK-Attention.

For T-MLP, the model initializes a hidden state from the last time step and then walks backward through time. At each earlier step, a sigmoid gate decides how much of the future hidden state to keep versus how much of the current input to inject, and the result passes through a LIF neuron. This replaces the standard MLP up-projection rather than adding a separate module, keeping the down-sampling structure intact.

Temporal fusion is applied only in the shallow layers (the first 50% of Transformer layers, i.e., the first two of four blocks), based on the reasoning that shallow layers capture low-level spatiotemporal patterns where fusion helps most, while deeper layers handle semantic refinement. All models use 4 Transformer blocks with embedding dimension 384 and are trained for 400 epochs on a single NVIDIA Tesla A100 GPU. The design also uses QK-Attention in shallow layers for efficiency and SSA in deeper layers for finer spatiotemporal modeling.

Why This Matters

Research impact. The paper argues that temporal modeling in Spiking Transformers has been under-explored, with prior methods like TIM, STAtten, and ST-SSA relying on heuristic hyperparameters or step-wise sequential processing that compromises parallelism. TEFormer offers a principled alternative and, according to the authors, the first systematic comparison of Spiking Transformers across neuron encoding schemes, suggesting encoding-based evaluation as a diagnostic paradigm.

Real-world applications:

• Event-driven vision from dynamic vision sensors (DVS), such as the CIFAR10-DVS, N-CALTECH101, and NCARS benchmarks.
• Event-based action recognition and video classification, as tested on HMDB51-DVS and UCF101-DVS.
• Spoken-digit and speech recognition using spike-based audio, as tested on SHD.
• Low-power neuromorphic hardware deployment, given the energy estimates reported (for example, 2.39 mJ total at MLP ratio 2).

Industry relevance. Neuromorphic processors and event-driven sensing are cited as emerging substrates for spiking computation. The paper reports that at MLP ratio 2, TEFormer uses less estimated energy than Spikformer, QKFormer, and TIM while achieving higher accuracy, which positions it for energy- or compute-constrained settings. The authors note that all results are from software simulation; hardware deployment is listed as unexplored.

Future Directions

• Hardware deployment: The authors state that results are based on software simulation only, so mapping TEFormer to neuromorphic hardware remains open.
• Beyond classification: Extension to more complex tasks such as segmentation and detection is named as unexplored, even though prior Spiking Transformers have been applied to those areas.
• Encoding standards: The paper notes that the SNN community still lacks a unified neuron encoding standard, despite encoding choices being influential for temporal modeling.
• Online and causal inference: Because T-MLP uses backward temporal fusion, TEFormer targets window-based recognition rather than strictly causal online inference, an acknowledged limitation for streaming applications.

Target Audience

Researchers and graduate students working on Spiking Neural Networks, Spiking Transformers, and neuromorphic computing who are already comfortable with LIF neuron dynamics, spiking self-attention, and standard vision and event-based benchmarks. It is also relevant to practitioners evaluating energy-conscious sequence models, though the paper reports simulation-based estimates only and no hardware results.

Authors’ abstract

In recent years, Spiking Neural Networks (SNNs) have achieved remarkable progress, with Spiking Transformers emerging as a promising architecture for energy-efficient sequence modeling. However, existing Spiking Transformers still lack a principled mechanism for effective temporal fusion, limiting their ability to fully exploit spatiotemporal dependencies. Inspired by feedforward feedback modulation in the human visual pathway, we propose TEFormer, the first Spiking Transformer framework that achieves bidirectional temporal fusion by decoupling temporal modeling across its core components. Specifically, TEFormer employs a lightweight and hyperparameter-free forward temporal fusion mechanism in the attention module, enabling fully parallel computation, while incorporating a backward gated recurrent structure in the MLP to aggregate temporal information in reverse order and reinforce temporal consistency. Extensive experiments across a wide range of benchmarks demonstrate that TEFormer consistently and significantly outperforms strong SNN and Spiking Transformer baselines under diverse datasets. Moreover, through the first systematic evaluation of Spiking Transformers under different neural encoding schemes, we show that the performance gains of TEFormer remain stable across encoding choices, indicating that the improved temporal modeling directly translates into reliable accuracy improvements across varied spiking representations. These results collectively establish TEFormer as an effective and general framework for temporal modeling in Spiking Transformers. Code is available https://github.com/Fancyssc/TEFormer.