Spiking Neural Networks for Continuous Control: Neuromorphic Reinforcement Learning in Conventional Computing

AI summary

Overview

• Research area: Neuromorphic reinforcement learning; specifically, spiking neural networks (SNNs) used as the actor in a continuous-control RL algorithm.
• Technical level: Intermediate. The paper assumes familiarity with reinforcement learning (policy gradients, actor-critic, replay buffers) and with basic SNN concepts (spike trains, membrane dynamics, surrogate gradients). It does not assume prior neuromorphic hardware experience.
• Scope in one sentence: The paper introduces SANSAC, a Soft Actor-Critic variant whose actor network is replaced by a spiking neural network, and systematically compares it against standard SAC on identical conventional hardware across four hidden-dimension settings in the Bipedal Walker environment.

What This Paper Is About

Neuromorphic chips promise large energy and parallelism advantages for reinforcement learning, but it is not yet clear whether swapping a conventional actor network for a spiking one changes how well an agent actually learns, before any hardware benefit appears. The authors build a minimal spiking actor variant of Soft Actor-Critic (SANSAC) and run it on ordinary GPU hardware side-by-side with standard SAC, so that any performance difference is attributable to the spiking design rather than to the chip it runs on. The goal is a reproducible baseline that future neuromorphic implementations can be measured against.

Key Contributions

1. A minimal spiking actor variant of SAC. SANSAC keeps SAC's two deep critic networks, replay buffer, and loss functions unchanged, and replaces only the actor with a spiking neural network — a modification designed to be deployable on neuromorphic hardware while training remains hybrid.
2. A systematic same-hardware comparison. Rather than claiming a new algorithm, the paper benchmarks SANSAC against SAC on identical conventional hardware across four hidden-dimension configurations (256, 200, 128, and 64 units), isolating the effect of hidden dimension on policy learning.
3. Statistical and practical characterization. The comparison reports mean return, standard deviation, success rate, Mann-Whitney U p-values, Cohen's d effect sizes, and wall-clock training time, concluding statistical equivalence alongside a measurable training-time penalty for the spiking actor.
4. A reproducibility baseline with public code. The implementation is released publicly, and the authors position the results as a baseline that future neuromorphic RL work can be compared against, since current SNN-versus-DNN comparisons for continuous control on conventional computers are sparse.

Main Findings

• Near-equivalent task performance. SANSAC achieved performance statistically indistinguishable from SAC across all four hidden-dimension configurations tested.
• No significant statistical differences. Mann-Whitney U tests gave p-values of 0.7913, 0.6232, 0.6776, and 0.5205 across the four settings, all above 0.05. Cohen's d effect sizes were 0.080, -0.140, 0.120, and -0.180, which the paper describes as negligible to small.
• Reported returns and success rates. Across the four configurations, the paper reports mean returns of 273.9 ± 130.8 / 224.5 ± 162.9; 197.3 ± 178.2 / 259.8 ± 146.2; 210.9 ± 185.9 / 158.7 ± 213.5; and 26.8 ± 149.4 / 41.8 ± 138.5, with success rates (agents reaching reward ≥ 300) of 70% / 80%, 60% / 80%, 70% / 60%, and 10% / 10%. The extracted table text does not make clear which member of each pair belongs to SANSAC versus SAC, or which row maps to which hidden dimension.
• Hidden dimension matters. Both algorithms degrade sharply at the smallest network setting, where only 10% of agents solved the task, and both improve substantially as hidden dimension increases. The paper treats the highest hidden dimension setting as its standard baseline because it produced the most stable learning.
• SANSAC trains roughly twice as slowly. Average training time was 61 minutes for SANSAC versus 21 minutes for SAC across the reported configurations, with per-configuration values of 44 vs 10, 66 vs 26, 80 vs 23, and 54 vs 24 minutes. The authors attribute this to the temporal unrolling that spiking dynamics require on sequentially operating conventional hardware.
• Higher variance in the spiking actor. The paper reports that SANSAC showed overall higher variance than SAC in two of the configurations, and notes instability in SANSAC's reward curve appearing as reward drops relative to SAC.
• Energy use not reported. The authors explicitly do not document an energy comparison between SANSAC and SAC on conventional hardware, stating that their measurements were inconsistent and hard to isolate.
• Curve-level similarity. Reward and actor loss curves over 10 random seeds show similar convergence behavior and shapes for both algorithms, suggesting SANSAC learns comparable behavior to SAC despite the different neuron model.

Methodology in Plain English

The authors take Soft Actor-Critic — a stochastic, entropy-maximizing algorithm that has performed well in the Farama Foundation Gymnasium Bipedal Walker environment — and change one component. Standard SAC uses one actor network and two critic networks, each with three fully connected layers. In SANSAC, the two critics stay as ordinary deep networks, but the actor becomes a spiking neural network built from Leaky-Integrate-and-Fire (LIF) neurons, with the final output layer changed to non-spiking Leaky-Integrate (LI) neurons so that the network produces a continuous voltage rather than a spike train, which is necessary for a continuous action space.

Because spikes are non-differentiable, training uses a sigmoid surrogate gradient, which lets ordinary backpropagation proceed through time over the unrolled network — implemented in the SpikingJelly library. The authors derived a numerically stabler form of the SAC policy log-probability, replacing tanh terms with the softplus function to reduce overflow during backpropagation. No population encoding or decoding is used, a deliberate choice to keep the comparison clean and to avoid added complexity. Since full neuromorphic deployments would discard the critics and replay buffer, only the actor matters after training.

Both algorithms were trained for up to 1200 episodes with identical early stopping: training halted if episodic reward did not improve for 100 consecutive episodes after at least 50 episodes. Seeds were generated randomly and kept in sync between SAC and SANSAC. Four hidden-dimension settings were tested — 256, 200, 128, and 64 units — applied identically to actor and critic networks in both algorithms. Experiments ran on an Nvidia GeForce RTX 4070. Key hyperparameters: learning rate 3·10⁻⁴, discount 0.975, replay buffer size 10⁶, minibatch of 256 samples, target smoothing coefficient 0.005, reward scale 5, 4 action dimensions, 24 state dimensions, alpha 0.2, 16 SANSAC timesteps, and the Adam optimizer.

Why This Matters

Impact on research. Most claims about SNN efficiency depend on neuromorphic hardware that is hard to obtain, so the community lacks a clean picture of whether the spiking design itself costs anything in learning quality. This paper isolates that question by holding hardware constant, and its finding of statistical equivalence — combined with a doubled training time — gives neuromorphic RL researchers a concrete, reproducible reference point and a clear statement of what remains unmeasured (energy, and performance on actual spiking chips).

Real-world applications:

• Robotics: Bipedal locomotion, the testbed here, is representative of legged and mobile robot control, where low-power embedded decision-making matters.
• Embedded and edge control systems: Devices with tight power and space budgets, such as drones or remote sensors, that need in-situ learning rather than cloud compute.
• Wearables and implantable medical devices: Continuous control under severe energy constraints, where event-driven computation has a natural fit.
• Autonomous sensory processing: Always-on perception and decision pipelines that would benefit from parallel, event-driven computation instead of sequential CPU/memory traffic.

Industry relevance. The paper is relevant to neuromorphic chip developers and to teams building RL-based control for power- or size-constrained hardware. The key caveat for adopters is that the demonstrated benefit is architectural potential, not measured efficiency: no energy savings are reported here, and the spiking actor was roughly 2x slower to train on conventional hardware.

Future Directions

1. Test alternative neuron models and gradient approximations. The current implementation uses LIF neurons and a sigmoid surrogate gradient; the authors suggest other neuron models and encoding/decoding schemes (including population encoding, deliberately excluded here) as paths to improved performance.
2. Deploy on physical neuromorphic hardware or FPGAs. Since most neuromorphic chips remain experimental and hard to acquire, the authors propose FPGAs as a more accessible and customizable intermediate target for validating the design outside simulation.
3. Assess scalability. Whether SANSAC holds up in more complex environments and larger network architectures, and whether its higher variance and sensitivity to early stopping worsen at scale, remains untested.
4. Explore hybrid training strategies. Splitting training phases between conventional and neuromorphic hardware could offset the computational overhead observed in this study, and the true energy profile of the fully neuromorphic agent is still an open question.

Target Audience

This paper is most useful to reinforcement learning researchers investigating neuromorphic deployment, neuromorphic hardware and chip designers looking for validated algorithmic workloads, and graduate students entering SNN-based RL who need a clear, reproducible baseline. Practitioners focused purely on maximizing control performance on conventional hardware will find less value, since the paper's contribution is a careful equivalence check rather than a new state-of-the-art method. Readers should be comfortable with actor-critic methods and basic spiking neuron mechanics to follow the methodology sections.

Authors’ abstract

Reinforcement learning (RL) algorithms have made strides over the past decade applying them to a wide range of problems and control tasks. However, the deployment of RL on neuromorphic hardware for continuous control tasks remains under-validated. Namely it is unclear whether replacing a conventional actor network with a spiking neural network (SNN) affects the performance of an agent before any hardware-specific benefits manifest. We provide a systematic validation of a minimal, neuromorphically viable spiking actor variant of Soft Actor-Critic (SAC) on conventional hardware, establishing a baseline for future neuromorphic RL research. In this paper, we propose the Spiking Actor Network Soft Actor Critic (SANSAC) to address the use of RL frameworks in continuous environments, designed as a framework that can be implemented on neuromorphic hardware. We compare a traditional Soft Actor Critic (SAC) network to SANSAC in a traditional computer. We demonstrate the near equivalent performance of SANSAC and SAC, while addressing the impact of hidden dimensions. Our results demonstrate the viability of SNN based algorithms in complex continuous environments, as well as competitive performance to traditional neural networks in traditional computers, providing a basis to continue exploring the use of SNNs in continuous RL frameworks.