A Closer Look at Knowledge Distillation in Spiking Neural Network Training

AI summary

Overview

Research area: Spiking neural network (SNN) training, specifically knowledge distillation (KD) from pre-trained artificial neural networks (ANNs) into SNNs for image classification.

Technical level: Advanced — the paper assumes familiarity with spiking neurons, surrogate gradient training, knowledge distillation, and class activation maps.

Scope in one sentence: The paper identifies two overlooked mismatches between teacher ANNs and student SNNs (raw feature distributions and raw logit distributions) and proposes two distillation strategies — Saliency-scaled Activation Map Distillation (SAMD) and Noise-smoothed Logits Distillation (NLD) — combined into a framework called CKDSNN.

What This Paper Is About

When an ANN is used as a teacher to train an SNN, existing methods simply line up the teacher's intermediate features and output logits with the student's, element by element. But ANN features are continuous floating-point values while SNN features are discrete binary spikes generated over multiple time steps, and SNN logits are sparser and more sharply peaked than ANN logits. The paper asks whether distillation can be made more effective by first making the two models' representations more comparable in semantics and in distribution, rather than forcing a match on raw, incompatible quantities.

Key Contributions

1. Saliency-scaled Activation Map Distillation (SAMD): Instead of aligning raw features, SAMD aligns the student SNN's Spiking Activation Map (SAM) with the teacher ANN's Class Activation Map (CAM), then applies a softmax-based "saliency scaling" so both maps become comparable probability distributions over pixels. The KL divergence between them becomes the SAMD loss.

2. Noise-smoothed Logits Distillation (NLD): The student SNN's logits are softened by adding Gaussian noise sampled with the mean and standard deviation of those same logits, then fused via a balancing hyper-parameter. This moves the sparse, peaked SNN logit distribution toward the denser, more continuous ANN logit distribution before a KL-based logits loss is applied.

3. A SAM generation method designed for SNNs: Because surrogate gradient estimation errors make gradient-based CAM generation (Grad-CAM style) inaccurate in SNNs, the authors directly compute the spike activation map by summing spike features across the channel and time dimensions. This is described as computationally efficient and performed online.

4. State-of-the-art results and compatibility: CKDSNN reports new best accuracies on CIFAR-10, CIFAR-100, ImageNet-1K, and CIFAR10-DVS, and the authors state the strategies can be flexibly integrated into existing KD approaches for SNN training.

Main Findings

• CIFAR-10/100 gains (ResNet-19, 1 time step): Compared with EnOF, CKDSNN improves CIFAR-10 accuracy by 0.74% and CIFAR-100 by 1.03%. The text states the resulting accuracies as 96.11% and 78.15%, but Table 1 lists 96.11% for CIFAR-10 and 79.11% for CIFAR-100 — the CIFAR-100 figure differs between the running text and the table, and 79.11 is also the value used in the ablation table, so the reported values are internally inconsistent.

• Accuracy improves with more time steps: On ResNet-19, CKDSNN reaches 96.11 / 79.11 (1 step), 97.13 / 83.21 (2 steps) and 97.81 / 83.88 (4 steps) on CIFAR-10 / CIFAR-100. On ResNet-20 it reaches 92.85 / 72.45 (1 step), 93.53 / 73.67 (2 steps) and 94.78 / 73.88 (4 steps). On the ViT-based setup (ViT-S teacher, Spikformer-4-384 student) it reaches 96.93 / 83.07 (1 step), 96.98 / 84.53 (2 steps) and 97.54 / 84.88 (4 steps).

• ImageNet-1K gains: CKDSNN reaches 66.92% top-1 with ResNet18 and 73.05% with ResNet34 at 4 time steps, versus BKDSNN's 65.60% and 71.24%. The text describes improvements of 1.32%, 1.81% and 1.52% across three network architectures (ResNet-18, ResNet-34, ResNet-50), but Table 2 shows only the ResNet18 and ResNet34 columns, so the ResNet-50 number is not visible in the provided content.

• Neuromorphic dataset gain: On CIFAR10-DVS with ResNet20 at 10 time steps, CKDSNN reaches 81.55%, an improvement of 1.05% over EnOFSNN (80.50%), and above BKDSNN (79.53%) and KDSNN (78.31%).

• Softmax scaling beats alternatives: In the SAMD ablation (ResNet-19, CIFAR-100), softmax scaling gives 79.11%, versus 76.48% for L2-norm, 75.56% with no scaling, and 74.78% for Z-score — a gap of roughly 2 to 3 points over the potential choices.

• Gradient-based CAM distillation degrades when applied to SNNs: Using ANN-oriented activation-map methods such as e²KD or CATKD for the SNN student performs "significantly lower" than the proposed CAM-SAM distillation, which the authors attribute to surrogate gradient estimation error; CATKD is additionally noted as CNN-only.

• Adaptive noise beats fixed noise: Small fixed noise gives poor results, increasing it improves performance but still trails the adaptive scheme, and excessive noise degrades performance, supporting the use of noise scaled to the SNN's own logit statistics.

• Energy efficiency improves: On ResNet34 / ImageNet-1k, CKDSNN at 4 time steps shows a fire rate of 13.0%, 4.01 G SOPS, 3.96 mJ and 73.05% accuracy, versus BKDSNN at 15.0%, 4.02 G, 3.98 mJ and 71.24%. At 2 time steps CKDSNN reaches 8.0%, 3.73 G, 3.61 mJ with 71.33% accuracy; at 3 time steps, 10.0%, 3.92 G, 3.88 mJ with 72.71%.

• Training overhead is intermediate: Per-epoch overhead (min/epoch) is 18.20 for KDSNN, 20.35 for BKDSNN, and 20.12 for CKDSNN, while CKDSNN achieves the best accuracy (73.05% vs 71.24% and 67.18%).

• Qualitative evidence: t-SNE visualizations of CIFAR-100 spike features show CKDSNN features are more separable and closer to the teacher's feature distribution; spike activation map visualizations show better alignment with teacher CAMs on objects such as a sharpener and a chainsaw.

Methodology in Plain English

The authors take an existing recipe — train a small SNN under the supervision of a pre-trained ANN — and change what the student is asked to imitate.

For the feature side, rather than copying the teacher's raw activations, they compute a class activation map from the teacher: a heatmap showing which image regions the teacher considers relevant to the predicted class, produced with the standard gradient-weighted approach. For the student, they cannot use the equivalent gradient-based heatmap because the surrogate gradients used to train SNNs make those gradients unreliable. So they instead build the student's heatmap directly from its spikes, summing the binary spike features over all channels and all time steps to get a single spatial map of where the SNN fired most.

The two heatmaps still live on different numeric scales, because one comes from continuous values and the other from counts of 0/1 spikes. A softmax (with a temperature constant) converts both into probability distributions over pixels, so they can be compared with a KL divergence — that comparison is the SAMD loss.

For the output side, the student's classification scores are sparse and peaked compared with the teacher's. To fix this, the authors sample Gaussian noise using the mean and standard deviation of the student's own logits, add a scaled amount of it to those logits, and pass the result through a temperature-scaled softmax. The KL divergence between this smoothed student distribution and the teacher's distribution is the NLD loss.

The three losses — standard cross-entropy on the ground truth, SAMD and NLD — are combined with balancing weights. Experiments use ResNet-19/20, ResNet-18/34 and a ViT-S teacher with a Spikformer student, on CIFAR-10, CIFAR-100, ImageNet-1K and CIFAR10-DVS, with the spiking neurons simulated in SpikingJelly.

Why This Matters

Impact on research: The paper reframes SNN knowledge distillation as a distribution-matching problem rather than an element-wise regression problem, and shows that the "obvious" transfer of ANN activation-map distillation tools to SNNs fails because of surrogate gradient error. It also provides evidence that SNN students can approach teacher-level feature separability while using far fewer spikes.

Real-world applications (as enabled by efficient SNN inference):

• Always-on edge sensing and monitoring, where event-driven computation reduces power draw.
• Neuromorphic vision for robotics and drones, where spiking cameras already produce event data.
• Wearable and implantable health devices that must run classification under tight energy budgets.
• Low-power keyword, gesture or activity recognition on battery-constrained hardware.

Industry relevance: The reported metrics — lower fire rate, SOPS and power alongside higher accuracy — are the trade-offs that matter for deploying models on neuromorphic chips. The authors also report training overhead per epoch, which matters for whether the method is practical to actually train.

Future Directions

• Resolving the reported-number inconsistencies: The CIFAR-100 accuracy at 1 time step appears as both 78.15% and 79.11%, and the ImageNet text mentions a ResNet-50 result that is not shown in Table 2; these need reconciliation.
• Extending beyond image classification: All evaluations are classification tasks (including the neuromorphic CIFAR10-DVS), while the introduction cites SNN work in natural language processing and multimodal tasks — whether SAMD and NLD transfer there is untested.
• Handling surrogate gradient error directly: Because the gradient error is the reason gradient-based CAM generation fails in SNNs, better gradient estimation could unlock stronger activation-map distillation.
• Scaling and architecture generality: The method is claimed to be architecture-agnostic and is tested on ResNets and a Spikformer; larger backbones, deeper SNNs and detection or segmentation heads remain open questions.
• Hyper-parameter robustness: The main text uses fixed values (T and τ = 2.0, λ = 0.1, β and γ = 1.0) and defers sensitivity analysis to supplementary material, leaving the generality of these settings across datasets unclear from the main paper.

Target Audience

Researchers and graduate students working on spiking neural networks, energy-efficient deep learning, or knowledge distillation; engineers evaluating neuromorphic deployment trade-offs between accuracy, spike rate and power; and readers already comfortable with surrogate gradient training and class activation maps who want to understand why naive ANN-to-SNN distillation is suboptimal.

Authors’ abstract

Spiking Neural Networks (SNNs) become popular due to excellent energy efficiency, yet facing challenges for effective model training. Recent works improve this by introducing knowledge distillation (KD) techniques, with the pre-trained artificial neural networks (ANNs) used as teachers and the target SNNs as students. This is commonly accomplished through a straightforward element-wise alignment of intermediate features and prediction logits from ANNs and SNNs, often neglecting the intrinsic differences between their architectures. Specifically, ANN's outputs exhibit a continuous distribution, whereas SNN's outputs are characterized by sparsity and discreteness. To mitigate this issue, we introduce two innovative KD strategies. Firstly, we propose the Saliency-scaled Activation Map Distillation (SAMD), which aligns the spike activation map of the student SNN with the class-aware activation map of the teacher ANN. Rather than performing KD directly on the raw %and distinct features of ANN and SNN, our SAMD directs the student to learn from saliency activation maps that exhibit greater semantic and distribution consistency. Additionally, we propose a Noise-smoothed Logits Distillation (NLD), which utilizes Gaussian noise to smooth the sparse logits of student SNN, facilitating the alignment with continuous logits from teacher ANN. Extensive experiments on multiple datasets demonstrate the effectiveness of our methods. Code is available~\footnote{https://github.com/SinoLeu/CKDSNN.git}.