STAER: Temporal Aligned Rehearsal for Continual Spiking Neural Network

AI summary

Overview

• Research area: Neuromorphic computing and continual learning — specifically Class-Incremental Learning (CIL) with Spiking Neural Networks (SNNs).
• Technical level: Advanced. The paper assumes familiarity with spiking neuron models (Leaky Integrate-and-Fire), surrogate gradients, experience replay, and dynamic time warping.
• Scope: One sentence — the paper proposes a replay-based continual learning framework that uses a differentiable Soft-DTW alignment loss plus temporal contraction/expansion of output logits to prevent catastrophic forgetting in a deep spiking ResNet19.

What This Paper Is About

Spiking neural networks process information through discrete spikes whose exact timing carries meaning, which should in principle make them well suited to learning a stream of tasks. In practice, when an SNN learns a new task, the precise spike timings that encoded earlier classes drift, and the network forgets them — a problem the authors call temporal misalignment, which standard spike-count losses do not penalize. STAER (Spiking Temporal Alignment with Experience Replay) aims to explicitly preserve the temporal structure of past responses during replay so that a deep SNN can match the accuracy of comparable non-spiking ("ANN") continual learning baselines.

Key Contributions

1. First SNN-CL method built on differentiable soft-DTW. The authors propose using Soft Dynamic Time Warping to explicitly align spike/logit timing between current and stored past model responses, targeting the temporal drift that drives forgetting.

2. A temporal expansion and contraction mechanism. For every buffered sample, three logit sequences are stored — at time windows T, T/2, and 2T — and the current network's T-step output is aligned against all three, mimicking biological memory processes in which recalled traces are compressed or dilated.

3. Demonstrated scalability to a deep spiking backbone. STAER is implemented on a ResNet19 SNN (rather than the shallow networks common in prior SNN-CL work) and evaluated on Sequential-MNIST and Sequential-CIFAR10 across multiple replay buffer sizes and two time-step settings.

4. State-of-the-art results plus a full ablation and forgetting analysis. The paper reports Final Average Accuracy (FAA) and Forgetting (FRG) under both CIL and TIL protocols, alongside ablations over the alignment weight β and the compression/dilation weights α1, α2.

Main Findings

• STAER leads the SNN baselines under CIL at T = 4. On Sequential-MNIST, STAER reaches 94.18%, 96.92% and 98.48% for buffers of 200, 500 and 5120 (best SNN value at every buffer size). On Sequential-CIFAR10 it reaches 51.19%, 65.68% and 83.53% at the same buffer sizes, again the best SNN results.

• Temporal alignment pays off as the time window grows. The paper reports the largest improvement on Sequential-CIFAR10 at T = 4 with respect to T = 2, specifically +6.56% at buffer size 500. Raising T from 2 to 4 has little effect on snn-ER, snn-DER or snn-DER++, but clearly benefits STAER.

• At T = 2 the picture is mixed on the harder dataset. STAER remains competitive on Sequential-MNIST (93.71%, 96.70%, 97.53%) and beats snn-ER, but on Sequential-CIFAR10 the advantage becomes less consistent for large buffers (47.88%, 59.12%, 78.30%).

• STAER still trails the ANN baselines on accuracy. The non-spiking ER, DER and DER++ models reach up to 99.06% (DER++, Sequential-MNIST) and 87.08% (DER++, Sequential-CIFAR10) under CIL, and JOINT upper bounds are 99.32% / 98.25% for the ANN and 98.92% / 89.80% for the SNN at T = 4. The paper's claim is that STAER "matches or outperforms" strong ANN baselines in the sense of closing the gap, not exceeding them everywhere.

• Forgetting is consistently lower for STAER at T = 4. Realized values: 2.14, 0.94 and 0.44 on Sequential-MNIST and 35.47, 29.20 and 11.45 on Sequential-CIFAR10 for buffers 200, 500, 5120 — the lowest of all SNN methods in every one of those configurations, and lower than the ANN models in the most constrained setting on Sequential-CIFAR10 (ANN ER 63.11, DER 55.85, DER++ 47.81 at buffer 200).

• Both temporal directions matter, and both together matter most. Disabling individual loss terms on Sequential-MNIST: snn-ER 87.90%, snn-ER + SDTW 90.15%, STAER without dilation 91.02%, STAER without compression 91.69%, full STAER 93.71%.

• Same-resolution alignment alone is weaker. Setting α1 = α2 = 0, which removes the contraction and expansion SDTW terms and keeps only T-to-T alignment, drops the FAA to 89.95% at β = 10⁻⁴.

• The alignment weight is brittle if tuned too high. The ablation sweeps β ∈ {10⁻⁴, 10⁻³, 5×10⁻³} and α1, α2 ∈ {0, 0.1, 0.25, 0.5, 0.75, 1.0} at T = 2 over 5 random seeds on Sequential-MNIST. At β = 10⁻⁴ the best configuration is 93.71% (α1 = α2 = 0.5) and the landscape is comparatively smooth; at β = 10⁻³ the best is 93.32% at (α1, α2) = (0.1, 0) but some configurations collapse to 24.69%; at β = 5×10⁻³ the peak recovers to 92.29% at α1 = α2 = 1.0.

• Under TIL, STAER nearly reaches the JOINT ceiling. With T = 4, STAER reaches up to 99.88% on Sequential-MNIST and up to 96.24% on Sequential-CIFAR10, against SNN JOINT upper bounds of 99.92% and 96.75%.

Methodology in Plain English

The network is a ResNet19 convolutional backbone in which the residual blocks use Leaky Integrate-and-Fire spiking neurons instead of ReLU activations. Because spiking thresholds are non-differentiable, the authors train with surrogate gradients — a hard threshold in the forward pass and an arctangent approximation in the backward pass — using the snnTorch implementation. A static input image is replicated across T time steps to create a temporal axis, and the network produces real-valued logits at every time step rather than spike counts, which makes ordinary loss functions usable.

Training follows the experience replay recipe. A buffer holds past inputs, their labels, and something extra: the past model's logit sequences for those inputs at three temporal resolutions — T, T/2 and 2T. Longer or shorter resolutions are produced simply by changing how many times the static image is replicated, so no separate networks or retraining are needed. At each new task, the batch is a concatenation of current-task samples and replayed samples, and the loss has two parts. The first is standard cross-entropy on the time-averaged logits. The second is the Temporal Alignment loss: the current network's T-step logits for a replayed sample are compared against each of the three stored sequences using Soft-DTW, a differentiable relaxation of dynamic time warping that tolerates shifts and stretching along the time axis. Those three terms are combined as a normalized weighted sum with weights α1 (contraction) and α2 (dilation), and the whole alignment term is scaled by β before being added to the cross-entropy loss. The contraction and expansion terms are computed on half of each mini-batch.

The evaluation uses two benchmarks, each of 10 classes split into 5 tasks of 2 classes. Sequential-MNIST uses 28×28 grayscale digits with 1 epoch per task at batch size 10; Sequential-CIFAR10 uses 32×32 RGB images with standard augmentation, 50 epochs per task at batch size 32. Replay buffers are 200, 500 or 5120. The ANN and SNN share the same ResNet19 width configuration — a 128-channel stem plus three residual stages with 3, 3 and 2 Basic Blocks and channel widths {128, 256, 512} — differing only in neuron model, so that ER, DER and DER++ (whose published versions used a ResNet18) can be compared fairly. The SNN uses γ = 0.5 and Vth = 1.0. Hyperparameters are α1 = α2 = 0.5, β = 10⁻⁴, Adam with learning rate 3×10⁻³ and a cosine scheduler. Success is measured by Final Average Accuracy and by Forgetting, the average gap between each task's best-ever accuracy and its accuracy after the final task.

Why This Matters

The work positions SNNs as a viable substrate for lifelong learning rather than a curiosity limited to shallow models and static datasets. Its main research impact is showing that temporal alignment — not just replaying samples or distilling logits — is a distinct and necessary ingredient in the spiking setting, since the ablation shows that adding the contraction/expansion terms changes accuracy from 87.90% to 93.71% on Sequential-MNIST. It also supplies a mechanism (storing logits at several time resolutions and matching them with a differentiable time-warping loss) that transfers naturally to any model producing time-series outputs.

Real-world applications implied by the paper's framing:

• Always-on edge sensors and neuromorphic hardware that must pick up new object or gesture classes over time without erasing old ones.
• Event-camera perception pipelines, where the input is already a stream of asynchronous events and spike timing is the native data format.
• Wearable and implantable biomedical monitoring, where the paper's biological plausibility argument (rate plus temporal coding, spike-timing-dependent plasticity) is most relevant.
• Robotics operating in non-stationary environments, where tasks change and the cost of retraining from scratch on-device is prohibitive.

Industry relevance: the results matter most to groups building energy-efficient inference on neuromorphic chips, where a spiking model that retains previous knowledge across deployment shifts avoids costly full retraining and avoids storing raw datasets for privacy-sensitive continual updates.

Future Directions

1. Scaling to stronger backbones. The authors state they plan to extend STAER to transformer-based continual learners and incremental fine-tuning of composable large models, drawing on continual ViT techniques such as dynamic token expansion and prompt-based adaptation.

2. More realistic evaluation streams. They aim to evaluate on Multi-domain Task Incremental Learning (MTIL) and MTIL-style continual adaptation of vision-language models, as well as on event-based neuromorphic datasets rather than the statically encoded MNIST and CIFAR10 used here.

3. The T = 2 / large-buffer gap on Sequential-CIFAR10. STAER's advantage over snn-DER and snn-DER++ is not consistent there, and the paper does not report a per-dataset breakdown of why alignment helps less at shorter time windows — an open question.

4. Robustness of the alignment weight. β = 10⁻³ produced collapses as low as 24.69% in the ablation, so understanding and stabilizing the brittleness of the alignment objective at higher weights is unresolved. The paper also does not report standard deviations for the main CIL/TIL tables, wall-clock or energy costs, or ablations on Sequential-CIFAR10.

Target Audience

Researchers and graduate students working on continual learning, spiking neural networks, or neuromorphic computing who already understand LIF dynamics and replay-based CIL; practitioners evaluating whether a spiking backbone can replace an ANN backbone in an incremental deployment setting; and readers interested in biologically motivated learning mechanisms such as compressed replay and dual rate/timing coding who want to see them implemented and ablated quantitatively. The code is available at https://github.com/matteogianferrari/staer.

Authors’ abstract

Spiking Neural Networks (SNNs) are inherently suited for continuous learning due to their event-driven temporal dynamics; however, their application to Class-Incremental Learning (CIL) has been hindered by catastrophic forgetting and the temporal misalignment of spike patterns. In this work, we introduce Spiking Temporal Alignment with Experience Replay (STAER), a novel framework that explicitly preserves temporal structure to bridge the performance gap between SNNs and ANNs. Our approach integrates a differentiable Soft-DTW alignment loss to maintain spike timing fidelity and employs a temporal expansion and contraction mechanism on output logits to enforce robust representation learning. Implemented on a deep ResNet19 spiking backbone, STAER achieves state-of-the-art performance on Sequential-MNIST and Sequential-CIFAR10. Empirical results demonstrate that our method matches or outperforms strong ANN baselines (ER, DER++) while preserving biologically plausible dynamics. Ablation studies further confirm that explicit temporal alignment is critical for representational stability, positioning STAER as a scalable solution for spike-native lifelong learning. Code is available at https://github.com/matteogianferrari/staer.