A flexible framework for structural plasticity in GPU-accelerated sparse spiking neural networks

AI summary

Overview

Research area: Computational neuroscience and neuromorphic computing — specifically GPU-accelerated simulation of spiking neural networks (SNNs) and the simulation of structural plasticity (the creation and removal of synapses).

Technical level: Advanced. The paper assumes familiarity with CUDA/GPU parallelism, sparse connectivity data structures, spiking neuron models, and gradient-based learning rules for SNNs (e-prop, STDP).

Scope: The paper introduces a new GeNN primitive, "custom connectivity updates," that lets users define GPU-accelerated structural plasticity rules, and demonstrates it on two applications — DEEP R-rewired sparse SNN classifiers trained with e-prop, and a synaptic rewiring model of topographic map formation.

What This Paper Is About

Most SNN and ANN research deals with synaptic plasticity: changing the strength of connections that already exist. Biological brains also continually create and remove synapses (structural plasticity), and pruning is used in machine learning to shrink trained models — but mainstream ML frameworks are built for dense connectivity, so pruning does not reduce training cost. This paper presents a flexible, code-generation-based framework inside the GeNN simulator for defining structural plasticity rules that run efficiently on GPU-accelerated sparse networks, and shows it can train sparse classifiers and grow topographic maps.

Key Contributions

1. A structural plasticity framework for general-purpose parallel hardware. The authors describe it as what they believe is the first structural plasticity framework implemented on GPUs, designed to satisfy three stated constraints: connectivity data structures must not need re-allocation when connections are added or removed, the data moved per connection change must be minimal, and connectivity must remain efficiently indexed and readable (i.e., via coalesced GPU memory reads).

2. A new GeNN "custom connectivity update" model primitive. Users define rules in Python via pygenn.create_custom_connectivity_update_model, supplying a GPU-parallel section executed across rows of the synaptic matrix (row_update_code) and a serial host section (host_update_code). Rules can access neuron and synapse variables through variable references, use gennrand random number functions, add synapses with add_synapse, and iterate over or remove existing synapses with for_each_synapse / remove_synapse. Rules can also define their own additional pre_vars, post_vars, and vars.

3. A memory-efficient DEEP R implementation. Rather than representing sparse connectivity as an additional dense weight matrix (as in the original DEEP R), the implementation adds only two bitfield data structures on top of the standard GeNN sparse ragged matrix — one tracking the fixed sign of each potential connection and one tracking its presence. DEEP R is integrated into mlGeNN and invoked through EPropCompiler using deep_r_conns and deep_r_l1_strength (example value used: 0.005).

4. A parallelized synaptic rewiring algorithm for topographic maps. The authors develop a GeNN-compatible variation of the rewiring model of [28], inspired by a SpiNNaker implementation [11]. It combines distance-dependent formation with weight-dependent elimination, distributes rewiring attempts across presynaptic neurons on the host, uses a bitfield to mark candidate synaptic slots so each slot is considered at most once, and — unlike earlier implementations [28, 11] — does not impose a fixed maximum in-degree and does not permit multapses.

Main Findings

• Up to 10x reduction in training time: Compared to baseline dense models, the sparse classifiers reduce training time by up to 10 times.

• 90x fewer parameters with no accuracy loss: Combining e-prop with DEEP R trained very sparse networks on N-MNIST and DVS gesture classification "with no loss of accuracy while using 90 times fewer parameters than comparable recurrent dense SNN models."

• Structural plasticity is cheap: The time overhead of applying structural plasticity is less than 1%.

• Sparse models train an order of magnitude faster: The sparse models used are reported to be an order of magnitude faster to train than their dense counterparts.

• Constant sparsity during training: Unlike typical incremental pruning, DEEP R maintains a constant level of connection sparsity throughout training, removing synapses marked dormant (weight updates that would have changed their sign) and adding an equal number of randomly chosen replacements.

• Topographic maps form in faster-than-realtime simulation: The topographic map model, in which activity-independent Gaussian-distance formation is combined with activity-dependent STDP-driven elimination, refines receptive fields and embeds learnt input structures into the network connectivity. Although the large part of simulation time is spent on connectivity updates, the simulation is still faster than the previous implementation [11] and supports network upscaling experiments to test scaling performance.

• Not reported in the provided text: Specific accuracy values, per-dataset error rates, absolute training wall-clock times, and the numerical results of the simulation-speed-versus-network-size measurements are not included in the excerpt available; only the relative claims above are given. The upscaling range for the topographic map is parameterized by a scaling factor s drawn from {0, ..., 7}, with domain length L0 = 16 and network size N0 = 256 nodes per layer at s = 0.

Methodology in Plain English

Sparse connectivity as a ragged matrix. GeNN stores sparse connectivity as a ragged matrix: each row holds the postsynaptic target indices of one presynaptic neuron, and rows are padded to the largest out-degree in the matrix. Parallel 2D arrays hold synaptic variables such as weights and delays. To propagate spikes, one CUDA thread is assigned to each column; each warp of 32 adjacent threads performs coalesced reads of synaptic weights and accumulates input to postsynaptic neurons using atomic operations (handled efficiently by the L2 cache on modern GPUs). Because synapse order within a row is unimportant, new synapses can be appended to the end of a row and a removed synapse can be replaced by moving the last synapse in the row into its place and decrementing the out-degree — so no re-allocation is needed if extra padding is added to each row.

Defining plasticity rules in code strings. Building on GeNN's existing approach of letting users define neuron models and learning rules from Python strings containing GeNNCode (a subset of C99), the authors add custom connectivity updates. A rule has a GPU section run in parallel across rows and a CPU section run serially on the host, with access to random number generation, synaptic variables, and pre/post neuron variables.

DEEP R. After normal initialization, a custom connectivity update initializes two bitfields: all connectivity bits cleared, all sign bits randomized to fix the sign of every potential connection. Then it loops through initialized synapses, setting the corresponding connectivity bit and matching the sign bit to the initial weight's sign. After each training batch, three custom updates run: an L1 regularization update that adds or subtracts a constant from e-prop's accumulated gradients based on the sign bitfield; an elimination update that removes synapses whose weight sign no longer matches the bitfield and counts dormant synapses per row; and a formation update that sums the dormant counts on the CPU, redistributes them uniformly as new activations across rows, and creates new synapses on the GPU while checking the connectivity bitfield to avoid duplicates. The additional noise term from the original DEEP R is omitted, following [24].

Topographic map. The model has two square layers of nodes on a grid with periodic boundary conditions: a source layer of Poisson spike generators and a target layer of excitatory conductance-based LIF neurons, with feed-forward and lateral connections initialized from a spatial profile. Input firing rates follow a Gaussian around a stimulus center that moves randomly at regular intervals. Structural plasticity runs as a custom connectivity update at a fixed rewiring interval, triggering a fixed number of rewiring attempts distributed across presynaptic neurons on the host. On the GPU, each presynaptic neuron sets bits in its own section of a bitfield for its assigned attempts. Elimination then walks the existing synapses, removing those whose bits are set according to an elimination probability that is higher for depressed (weak) synapses than for potentiated ones. Remaining set bits indicate empty slots, and formation then iterates the bitfield row, creating a synapse with a Gaussian probability based on the lateral distance between the neurons. STDP varies synaptic strengths concurrently, with conductances restricted between a minimum and maximum. A transpose matrix used for postsynaptic spike-triggered updates is rebuilt after each custom connectivity update if the connectivity is affected.

Classifiers. Single recurrently connected hidden layer of 512 neurons, trained on N-MNIST and DVS gesture. DVS gesture examples are spatially downsampled from 128x128 to 32x32 and temporally downsampled so there is one spike per 1 ms simulation timestep. DVS gesture uses adaptive LIF (ALIF) neurons (membrane time constant 20 ms, baseline threshold 0.6, adaptation time constant 2000 ms, adaptation strength 0.0174); N-MNIST uses standard LIF neurons (obtained by setting beta = 0 and rho = 0). Input and recurrent weights are sparse and initialized with a fixed connection probability (5% given as an example). Readout uses simple leaky integrators, gradients come from e-prop eligibility traces, and all gradients are applied with the Adam optimizer.

Timing instrumentation. GeNN's existing kernel timing (CUDA events on GPU, std::chrono::high_resolution_clock on CPU) was extended to time the CPU and GPU portions of custom connectivity updates and the transpose remapping.

Why This Matters

The work matters because structural plasticity has been hard to study at scale: biological structural changes occur over hours or days and therefore demand long simulations, they interact with synaptic plasticity in complex ways, and continuously modifying connectivity conflicts with the usual practice of holding network structure fixed in memory during state propagation. Prior structural plasticity implementations have largely targeted CPU-based simulators and neuromorphic systems built on scalar cores (SpiNNaker) or a SIMD plasticity processing unit (BrainScaleS-2 PPU). This framework brings structural plasticity to general-purpose GPU accelerators and demonstrates a concrete payoff in training efficiency, not just biological fidelity.

Real-world applications:

• Training and deploying sparse SNN classifiers for event-based vision and gesture recognition on power-constrained hardware, since fewer parameters mean lower memory and compute requirements.
• Neuromorphic computing pipelines that need to maintain sparsity during training and inference, rather than only pruning once after training.
• Computational neuroscience studies of development, learning, memory, lesion recovery, and sensory deprivation that require long-duration simulation of changing connectivity.
• Modeling of topographic map formation as a step toward understanding how spatial and functional organization arises in sensory systems.

Industry relevance: The paper speaks directly to the cost of training ever-larger models. Because mainstream backpropagation frameworks are optimized for dense connectivity, pruning does not help reduce training cost there. A framework that keeps networks sparse throughout training — while making rewiring overhead less than 1% of runtime and delivering up to 10x faster training with 90x fewer parameters at no accuracy loss — is relevant to anyone building efficient neuromorphic or event-driven inference systems, and to hardware and simulator developers targeting sparse neural communication.

Future Directions

• Extending the framework to additional structural plasticity rules. The authors reviewed the literature to enumerate required functionality (access to synaptic weights, extra pre/post variables, access to pre- and postsynaptic neuron variables, and access to random number generation), implying further rules remain to be implemented and tested.
• Reducing the cost of connectivity updates. In the topographic map simulation, the large part of simulation time is spent on connectivity updates, which the authors flag as the target for upscaling performance.
• Scaling studies. The paper measures simulation speed versus network size and enables network upscaling experiments (parameterized by scale factors 0 through 7 in the model description), suggesting further scale testing is expected.
• Applying the work across neuromorphic platforms. The authors state the framework will enable further research into achieving and maintaining sparsity in network structure and neural communication, and "exploring the computational benefits of sparsity in a range of neuromorphic applications."
• An AMD HIP backend is noted as under development, opening the door to non-NVIDIA GPU execution.

Target Audience

This paper is most useful to SNN simulator developers and computational neuroscientists who study structural plasticity and topographic map formation, and to machine learning researchers interested in training sparse networks efficiently. It will also interest neuromorphic hardware and systems engineers, since the framework targets GPU accelerators that sit alongside platforms such as SpiNNaker and BrainScaleS-2. Readers should be comfortable with GPU parallelism, sparse matrix representations, and spiking neuron and plasticity models; the paper's code examples are in Python (PyGeNN/mlGeNN) with GeNNCode strings written in a C99 subset.

Authors’ abstract

The majority of research in both training Artificial Neural Networks (ANNs) and modeling learning in biological brains focuses on synaptic plasticity, where learning equates to changing the strength of existing connections. However, in biological brains, structural plasticity - where new connections are created and others removed - is also vital, not only for effective learning but also for recovery from damage and optimal resource usage. Inspired by structural plasticity, pruning is often used in machine learning to remove weak connections from trained models to reduce the computational requirements of inference. However, the machine learning frameworks typically used for backpropagation-based training of both ANNs and Spiking Neural Networks (SNNs) are optimized for dense connectivity, meaning that pruning does not help reduce the training costs of ever-larger models. The GeNN simulator already supports efficient GPU-accelerated simulation of sparse SNNs for computational neuroscience and machine learning. Here, we present a new flexible framework for implementing GPU-accelerated structural plasticity rules and demonstrate this first using the e-prop supervised learning rule and DEEP R to train efficient, sparse SNN classifiers and then, in an unsupervised learning context, to learn topographic maps. Compared to baseline dense models, our sparse classifiers reduce training time by up to 10x while the DEEP R rewiring enables them to perform as well as the original models. We demonstrate topographic map formation in faster-than-realtime simulations, provide insights into the connectivity evolution, and measure simulation speed versus network size. The proposed framework will enable further research into achieving and maintaining sparsity in network structure and neural communication, as well as exploring the computational benefits of sparsity in a range of neuromorphic applications.