Hardware-accelerated graph neural networks: an alternative approach for neuromorphic event-based audio classification and keyword spotting on SoC FPGA

AI summary

Overview

Research area: Neuromorphic computing and edge AI hardware — specifically FPGA acceleration of event-graph neural networks for event-based audio processing.

Technical level: Advanced. The paper assumes familiarity with graph neural networks, quantisation, FPGA design flows (BRAM, programmable logic vs. processing system), spiking neural networks, and neuromorphic sensors.

Scope: The paper designs and evaluates a hardware-accelerated graph convolutional network on a Xilinx ZCU104 SoC FPGA for two tasks — event-based audio classification (Spiking Heidelberg Digits, Spiking Speech Commands) and end-to-end keyword spotting — and compares it against prior FPGA-based spiking neural network implementations.

What This Paper Is About

Most event-based audio systems use spiking neural networks (SNNs), but SNN weight-memory access is non-deterministic, which makes it hard to actually exploit the sparsity of event data on hardware. This paper instead converts artificial-cochlea event streams into spectro-temporal graphs and processes them with a graph convolutional network whose access pattern is deterministic, then maps that model onto a SoC FPGA. The goal is an end-to-end, continuously running system for two tasks: classifying whole spoken digits/words from a single recording, and spotting keywords in an unsegmented event stream.

Key Contributions

1. Hardware-aware redesign of spectro-temporal event-graphs. The authors modify the reference event-graph model to fit reconfigurable hardware: a new graph generator using a 1D context memory indexed by cochlea channel, replacement of computationally expensive event-normal-vector features with averaged neighbour timestamp/channel coordinates, a deterministic "skip step connection" neighbour search, added batch normalisation, and an additional positional normalisation (PN) step on edge position differences. Model hyperparameters were also chosen through ablation studies.

2. The first embedded system for event-graph-based audio processing on a SoC FPGA, and the first hardware implementation supporting fully asynchronous event-by-event processing that preserves the temporal sparsity of the data.

3. New state-of-the-art results for time-series classification on an FPGA for the Spiking Heidelberg Digits (SHD) dataset, plus the first FPGA-based benchmark results on the Spiking Speech Commands (SSC) dataset.

4. The first end-to-end hardware implementation of a keyword spotting model on event-based audio, combining graph convolutional layers with recurrent (GRU) sequence modelling and designed for continuous, real-time operation.

Main Findings

• Classification accuracy on SHD: The baseline floating-point model achieves 92.7% accuracy, which the authors state is 2.4% below the state of the art, while requiring over 10× and 67× fewer parameters.

• Accuracy on SSC: The models achieve 66.9–71.0% accuracy. The authors report this is the first hardware-accelerated evaluation on this dataset.

• Quantised model beats prior FPGA SNNs: The quantised model reaches 92.3% accuracy, outperforming FPGA-based spiking neural network implementations by up to 19.3%, while also reducing resource usage and latency.

• Keyword spotting performance: The end-to-end FPGA KWS system achieves up to 95% word-end detection accuracy with 10.53 μs latency and 1.18 W power consumption.

• Throughput analysis: For the classification baseline running at a 200 MHz clock, the theoretical throughput is 555 kEPS (thousand events per second), while the average event rate in the SHD dataset is around 20 kEPS.

• Pooling choice matters for classification, not for KWS: In classification, replacing global average pooling with max pooling caused a drop of 7 pps and sum pooling a drop of 5 pps, so average pooling was kept. In keyword spotting, where the aggregation window is much shorter (10 ms versus 1000 ms for classification), average/add and max pooling performed almost identically, with a slight edge for max pooling — which is also cheaper in hardware because it needs no accumulation or division.

• Prior FPGA SNN baselines: The Spiker+ framework achieved 72.9% on SHD at 430 mW and 540 μs latency; QUANTISENC reported 87.8% accuracy with 1.6 W peak power consumption.

• Full resource-utilisation and power tables are not present in the provided content of the paper (the text is truncated), so specific LUT/DSP/BRAM figures and the measured power/latency breakdown for the SHD classification task are not reported here.

Methodology in Plain English

An artificial cochlea converts sound into sparse asynchronous events: each event has a channel number (roughly a frequency band, since the cochlea splits audio into channels with band-pass filters) and a timestamp. Normally these events would be fed to a spiking neural network. Instead, the authors turn each new event into a node in a graph, connecting it to previously stored events that fall within a channel radius r_ch = 100 and a time radius r_t, forming a directed graph.

To make this cheap in hardware, they store only the most recent event per channel in a small block-RAM table (the channel index is the address, the timestamp is the data), so each newly arriving event has at most one neighbour per channel and memory stays tiny. Neighbour search uses a fixed "skip step" pattern (skip step s = 10) rather than scanning every channel, and neighbour features are summarised by simple averages of timestamp and channel coordinates instead of fitting local surfaces.

The resulting graph passes through four graph convolution (PointNetConv-style) layers. Each layer takes the event's own features plus position differences to its neighbours, normalises those position differences into the (0,1) range, applies a small fully connected transform, and then max-pools over neighbours with a ReLU. Two-port BRAMs hold the last processed feature per channel and the weights, and processing is fully asynchronous and event-by-event, so latency is low and bursty event rates do not stall the pipeline. All arithmetic uses integers with per-layer selectable bit precision; batch normalisation layers are folded into the weights at quantisation time so they add no parameters.

From there, the two tasks diverge. For classification, features of all events in a sample are accumulated in a register and divided by an event counter (global average pooling), then passed to a small MLP with two ReLU layers and a softmax — implemented either on the processing system in floating point or in programmable logic with 8-bit quantisation, and the two options are compared. For keyword spotting, the stream is chopped into 10 ms windows, max-pooled per window, passed through a shallow two-layer STEM, and fed to a GRU that maintains a hidden state across windows; each step emits both class probabilities and a scalar confidence score, produced by a state-machine-controlled head that shares vector-multiplication units between the successive MLPs.

Why This Matters

Impact on research: The work tests whether the theoretical advantage of event-graphs over SNNs — deterministic weight access and natively asynchronous operation — actually holds up when implemented in silicon. The reported accuracy, resource and latency improvements over FPGA SNN baselines on SHD, plus the first FPGA results on SSC and the first end-to-end event-audio KWS on FPGA, give the community concrete reference points for a non-SNN path to event-driven audio hardware.

Real-world applications:

• Always-on voice assistants and wake-word detection on battery-powered wearables and microcontrollers.
• Low-latency audio event monitoring in smart-home and industrial IoT sensor nodes.
• Robotics, where on-board spoken commands must be recognised without cloud round trips.
• Edge health or safety monitoring devices that must react to sound triggers under strict energy budgets.

Industry relevance: The system is built on a commercial SoC FPGA (Xilinx ZCU104) and uses standard quantisation tooling ideas (integer arithmetic, folded batch norm), so it is a plausible template for vendors building always-on audio front-ends. The reported 1.18 W and 10.53 μs latency for KWS are the kinds of figures hardware architects use to decide between FPGA, ASIC and microcontroller deployments.

Future Directions

• Scaling parallelism: The authors note that the graph convolution module is the bottleneck and that the number of parallel vector-multiplication modules can be increased to trade resources for latency and throughput, or decreased for the opposite effect — the optimal operating point across tasks is left open.
• Extending beyond binary word-end detection: The keyword spotting head emits a class and a confidence per 10 ms window; how well the first-hardware KWS benchmark generalises to larger vocabularies and noisier continuous speech is not yet established.
• Broadening dataset coverage: SSC is only the first hardware-accelerated evaluation, and the authors position it as a reference point; further FPGA comparisons and results on other event-audio benchmarks are natural next steps.
• Further hardware-aware model optimisation: The paper's own recipe — simplifying graph generation, adding positional normalisation, tuning hyperparameters through ablation — suggests more aggressive co-design of the event-graph representation and the accelerator is still available.

Target Audience

Researchers and engineers working on neuromorphic hardware, FPGA-based AI accelerators, and edge audio processing; practitioners designing always-on keyword spotting or voice-triggered embedded systems; and anyone comparing graph neural network approaches against spiking neural networks for event-driven sensor data. Readers need a working knowledge of neural network quantisation and digital hardware design to follow the architectural details.

Authors’ abstract

As the volume of data recorded by embedded edge sensors increases, particularly from neuromorphic devices producing discrete event streams, there is a growing need for hardware-aware neural architectures that enable efficient, low-latency, and energy-conscious local processing. We present an FPGA implementation of event-graph neural networks for audio processing. We utilise an artificial cochlea that converts time-series signals into sparse event data, reducing memory and computation costs. Our architecture was implemented on a SoC FPGA and evaluated on two open-source datasets. For classification task, our baseline floating-point model achieves 92.7% accuracy on SHD dataset - only 2.4% below the state of the art - while requiring over 10x and 67x fewer parameters. On SSC, our models achieve 66.9-71.0% accuracy. Compared to FPGA-based spiking neural networks, our quantised model reaches 92.3% accuracy, outperforming them by up to 19.3% while reducing resource usage and latency. For SSC, we report the first hardware-accelerated evaluation. We further demonstrate the first end-to-end FPGA implementation of event-audio keyword spotting, combining graph convolutional layers with recurrent sequence modelling. The system achieves up to 95% word-end detection accuracy, with only 10.53 microsecond latency and 1.18 W power consumption, establishing a strong benchmark for energy-efficient event-driven KWS.