Unveiling the Spatial-temporal Effective Receptive Fields of Spiking Neural Networks

AI summary

Overview

Research area: Neuromorphic computing / spiking neural networks (SNNs) for computer vision, specifically visual long-sequence modeling (object detection and semantic segmentation).

Technical level: Intermediate to Advanced. The paper leans on gradient-based receptive field theory, spiking neuron dynamics, and Transformer architecture design.

Scope: The paper introduces a Spatio-Temporal Effective Receptive Field (ST-ERF) framework for analyzing SNNs, uses it to diagnose a global-context limitation in Transformer-based SNNs, and proposes two channel-mixer modules (MLPixer and SRB) that improve dense prediction performance. arXiv:2510.21403v1 [cs.NE], 24 Oct 2025.

What This Paper Is About

Spiking Neural Networks are energy-efficient but lag behind conventional artificial neural networks (ANNs) on visual long-sequence modeling tasks, which require dense, spatially detailed outputs across an entire image. The standard effective receptive field (ERF) tool used to diagnose ANN feature extraction cannot be applied directly to SNNs because spiking neurons carry temporal dynamics. The authors build an extended, time-aware version of the ERF, use it to show that existing Transformer-based SNNs never form a truly global receptive field, and then design architectures that fix this.

Key Contributions

1. The ST-ERF framework. The authors formally extend the conventional ERF to the temporal dimension, defining it as the partial derivative of an output spike feature at time t with respect to an input spike feature at an earlier time t − τ. They also define a spatial ERF (a weighted average of ST-ERFs over timesteps) and a temporal ERF (an integration over spatial dimensions), and give a loss-derived method for computing both with PyTorch automatic differentiation rather than an explicit loss function.

2. A diagnosis of existing Transformer-based SNNs. Applying ST-ERF to multiple models, they report that current designs fail to establish a global ERF across all timesteps, which they attribute largely to convolution-based channel mixers that carry a locality bias.

3. Two new channel-mixer architectures. MLPixer replaces the convolutional channel mixer entirely with a two-layer MLP pipeline of batch normalization and spiking neuron layers; SRB (splash-and-reconstruct block) replaces only the second convolution in the mixer with a single-layer MLP, keeping a 1x1 convolution in the first position. Both keep spatial features separate during channel mixing.

4. Integration and validation. The modules are dropped into the first two stages of the Meta-SDT architecture (and into CAFormer / Meta-SDT style multi-stage designs), and evaluated on COCO 2017, ADE20K, and event-based tracking on FE108 and VisEvent.

Main Findings

• Transformer-based SNNs have narrow, center-heavy receptive fields. Compared with ViT-B, SDT-V1 shows a spatial ERF that is more centrally concentrated and markedly narrower. Meta-SDT behaves similarly to its ANN counterpart CAFormer-s18 but still struggles to maintain long-range feature attention; the authors link this to SDT-V1's Spike Patch Splitting module with its early convolutional layers and to Meta-SDT's Re-parameterization Convolution used for query/key/value projection.

• Spikformer behaves differently. In the authors' comparison across stages, Spikformer exhibits diffuse receptive fields across all stages, whereas SDT-V1, Meta-SDT, and QKFormer show markedly centered distributions that gradually expand with network depth, all manifesting a Gaussian-like effect. The authors also observed dissipation of the spatial ERF in SDT-V1 during the final stage.

• The proposed variants build global receptive fields early. MLPixer-SDT establishes a strong global spatial ERF already in Stage 1, then selectively contracts toward specific regions as depth increases. SRB-SDT only forms a preliminary spatial ERF at Stage 2, and this distribution continues to evolve with depth.

• Convolutions narrow the field; MLPs widen it. Visualizations show the vanilla convolution-based channel mixer has a limited ST-ERF, while MLPixer and SRB achieve a more global ST-ERF, with MLPixer broader than SRB because it uses fewer convolutions.

• COCO 2017 object detection results. Under the same 1x training schedule with Mask R-CNN, all proposed variants beat the SDTv3 baseline. SRB (ε=4) reaches 48.9 AP^b_50 with 37M parameters versus SDTv3-B's 46.9 AP^b_50 with 39M parameters, and full metrics of 25.8 AP^b, 22.8 AP^b_75, 22.5 AP^m, 43.9 AP^m_50, 20.4 AP^m_75 at 4 timesteps. For the tiny configuration, SRB (ε=4) reaches 39.2 AP^b_50 at 25M parameters versus SDTv3-T's 35.5 at 25M. The paper states SRB exceeds SDTv3-T and SDTv3-B by 10.42% and 4.26% on AP^b_50 respectively while maintaining almost the same model size.

• ADE20K semantic segmentation results. With Semantic FPN and ImageNet-1K pretraining over 160k iterations, SRB (ε=4) improves mIoU by 3.3% over SDTv3-T (38.2 vs 34.9) and 2.6% over SDTv3-B (43.7 vs 41.1), while reducing parameters by 0.3M and 1.2M respectively (6.2M and 19.2M). MLPixer (ε=4) achieves the largest parameter reduction of 0.6M and 2.4M (5.9M and 18.0M), with 34.9 and 42.0 mIoU.

• Event-based tracking results are mixed. On FE108, the proposed variants improve over SD-Track (Tiny) on AUC and PR: SD-Track reports 19.61M parameters, 56.7 AUC and 89.1 PR; +MLPixer (ε=4) gives 20.21M, 57.1 AUC, 89.2 PR; +MLPixer (ε=6) gives 22.99M, 57.9 AUC, 90.1 PR; +SRB (ε=4) gives 21.43M, 58.2 AUC, 88.5 PR. On VisEvent, all three variants score below SD-Track's 35.4 AUC and 48.7 PR (33.7/47.3, 34.5/48.9, 33.8/48.0). The authors write that these results confirm the designs preserve performance while highlighting opportunities for further improvement on temporal benchmarks.

• Analysis protocol. The ST-ERF experiments use 60 iterations with input tensors drawn from a standard normal distribution (μ=0, σ²=1), with the central spatial feature across all channels and timesteps set to uniform gradient stimuli of value 1, and the ERF averaged over all timesteps for visualization.

Methodology in Plain English

The authors start from the observation that a network's receptive field (the input region that can affect a given output) tells you about topology, but not about which inputs actually matter. The effective receptive field fixes this by measuring gradients: how much does the output change when you nudge each input? For spiking networks this is insufficient, because an output at time t can also depend on inputs from earlier timesteps.

Their ST-ERF therefore takes a derivative not just across space but also across time lags, producing a spatio-temporal map of influence. They show this can be computed cheaply without writing an explicit loss function: set the gradients flowing into the chosen output location to 1 and run ordinary back-propagation, then read off the gradients at the input. One gradient-pass recipe yields the spatial ERF (averaged over timesteps) and another yields the temporal ERF (summed over spatial positions).

Armed with this measurement tool, they probe several Transformer-based SNN architectures and observe that the receptive fields tend to be concentrated near the center and never become global, which they trace to convolutional operations in early stages and in the channel mixers. Their fix is architectural and simple: in the first two stages of a multi-stage SNN, swap the convolutional channel mixer for either a two-layer MLP mixer (MLPixer) or a hybrid that keeps one 1x1 convolution and replaces the rest with a single-layer MLP (SRB). Because MLPs act pixel-wise, they mix information across channels without imposing spatial locality, broadening the effective receptive field. The token mixer (a spike-driven separable convolution with pointwise and depthwise convolutions) and the membrane-shortcut residual connections are left in place. They then re-run the COCO, ADE20K, and event-tracking pipelines with these swapped-in backbones and compare against the published baselines under identical training schedules.

Why This Matters

Impact on research. The paper supplies SNN researchers with an analysis tool analogous to the ERF that ANNs have had since the gradient-based formulation, and it turns a vague observation ("SNNs underperform on dense prediction") into a concrete, measurable architectural property (absence of a global ST-ERF across timesteps). The result that pixel-wise MLP mixing broadens the receptive field more than convolution gives architecture designers a principled reason to revisit the widespread use of convolutional stem and mixer layers in spiking Transformers. The authors also state their belief that the ST-ERF framework can inform SNN design beyond the tasks tested.

Real-world applications.

• Autonomous driving and robotics perception, where object detection and instance segmentation on high-resolution frames are core workloads and energy budget matters.
• Event-camera vision systems, including the event-based tracking pipeline (SDTrack) the authors test, used in drones and high-speed motion capture where frames are sparse and asynchronous.
• Always-on or battery-limited edge vision, such as smart cameras and wearables, where the event-driven sparsity of SNNs is attractive.
• Semantic understanding of large scenes (the ADE20K-style task), relevant to mapping, agriculture, and remote sensing.

Industry relevance. Neuromorphic hardware vendors and edge-AI teams benefit from architectures that improve dense prediction accuracy without increasing parameter count; the SRB results on ADE20K specifically reduce parameters (0.3M and 1.2M fewer) while raising mIoU. The paper does not report energy or latency measurements for its own proposed modules in the provided content, so efficiency claims for MLPixer and SRB rest on parameter count and timestep settings rather than measured on-hardware energy.

Future Directions

• Extending the ST-ERF analysis beyond the tested architectures and tasks, including recurrent or state-space spiking models, and formalizing whether a global ST-ERF is necessary or merely sufficient for strong dense prediction.
• Understanding why MLPixer and SRB did not improve event-based tracking on VisEvent, where all three variants scored below SD-Track on AUC and PR; the authors themselves flag room for improvement on temporal benchmarks.
• Deriving the weighting function w(t, τ) for specific neuron models beyond the LIF discussion, so the spatial ERF's relation to the ST-ERF is exact for a given network.
• Applying the mixer redesign to the deeper stages and to other backbone families, since this work only replaces operations in the first two stages and leaves the Transformer-SNN blocks in Stages 3 and 4 configured as in Meta-SDT-V3.
• Quantifying energy and latency savings on neuromorphic hardware, which the provided content does not report for the proposed modules.

Target Audience

This paper is most useful to researchers and engineers working on spiking neural networks, neuromorphic computing, and energy-efficient computer vision, particularly those interested in interpretability and architecture design for dense prediction tasks. It also suits practitioners of receptive-field analysis and Transformer architecture design who want to understand how locality bias from convolutions affects spatio-temporal models. Readers need comfort with gradients, back-propagation, and Transformer block structure; the convolutional RPE, spike patch splitting, and re-parameterization convolution context assumes familiarity with the Spike-driven Transformer line of work. Readers looking for measured energy or latency results, or for a broad survey of SNN training methods, will need to look beyond this paper.

Authors’ abstract

Spiking Neural Networks (SNNs) demonstrate significant potential for energy-efficient neuromorphic computing through an event-driven paradigm. While training methods and computational models have greatly advanced, SNNs struggle to achieve competitive performance in visual long-sequence modeling tasks. In artificial neural networks, the effective receptive field (ERF) serves as a valuable tool for analyzing feature extraction capabilities in visual long-sequence modeling. Inspired by this, we introduce the Spatio-Temporal Effective Receptive Field (ST-ERF) to analyze the ERF distributions across various Transformer-based SNNs. Based on the proposed ST-ERF, we reveal that these models suffer from establishing a robust global ST-ERF, thereby limiting their visual feature modeling capabilities. To overcome this issue, we propose two novel channel-mixer architectures: \underline{m}ulti-\underline{l}ayer-\underline{p}erceptron-based m\underline{ixer} (MLPixer) and \underline{s}plash-and-\underline{r}econstruct \underline{b}lock (SRB). These architectures enhance global spatial ERF through all timesteps in early network stages of Transformer-based SNNs, improving performance on challenging visual long-sequence modeling tasks. Extensive experiments conducted on the Meta-SDT variants and across object detection and semantic segmentation tasks further validate the effectiveness of our proposed method. Beyond these specific applications, we believe the proposed ST-ERF framework can provide valuable insights for designing and optimizing SNN architectures across a broader range of tasks. The code is available at \href{https://github.com/EricZhang1412/Spatial-temporal-ERF}{\faGithub~EricZhang1412/Spatial-temporal-ERF}.