Bringing BNNs to Fast Event Processing

AI summary

Overview

• Research area: Computer Vision, specifically neuromorphic (event-based) vision combined with network quantization (Binary Neural Networks).
• Technical level: Advanced. The paper assumes familiarity with convolution, quantization, knowledge distillation, spiking networks, and operation counting (BOPs/FLOPs).
• Scope: The paper benchmarks state-of-the-art deep Binary Neural Networks on event-camera classification datasets, introduces a binary event representation called the Polar-wise Binary Event Volume (PBEV), and studies how event representation, accumulation window, and cross-modal RGB pretraining affect accuracy and computational cost.

What This Paper Is About

Event cameras record sparse, asynchronous streams of brightness-change events with low latency, high dynamic range and low power consumption, while Binary Neural Networks compress weights and activations to one bit to cut model size and inference cost. Despite both technologies being motivated by efficiency, the paper states their combination "remains largely unexplored." The goal is to adapt and evaluate modern deep BNN architectures on event data, measure them against a full-precision ResNet-18 baseline, and introduce a binary input representation that lets event data be processed by a fully binarized network.

Key Contributions

1. A new event representation: the Polar-wise Binary Event Volume (PBEV), which stacks short binary time-slices with each polarity in separate channels, producing a binary 3D tensor that binary networks can ingest directly. A PBEV with 2 channels is equivalent to the existing Binary Event Image.
2. An extensive benchmark of state-of-the-art deep convolutional BNNs on event-based classification, evaluated against a full-precision CNN baseline across multiple event representations and accumulation schemes, and positioned against published CNN, SNN and transformer results on N-Caltech101 and N-ImageNet-mini.
3. An analysis of what drives performance, covering the choice of event representation, the accumulation time window, and the effect of cross-modal pretraining from RGB weights on learning and generalization.
4. A complexity argument: all tested BNN architectures can equal or surpass the full-precision ResNet-18 baseline on event datasets while requiring from 7× up to 21× fewer operations.

Main Findings

• Best headline result: the best evaluated BNN on the N-Caltech101 classification benchmark reaches 90.58% accuracy with 7.5× fewer operations than its full-precision counterpart. The introduction and conclusion frame the broader range as 7× up to 21× fewer operations for the binary-input configurations.

• BNNs beat the full-precision baseline from scratch: distilled binary networks already match or exceed ResNet-18 on both datasets, with BNext Small leading on both. On N-Caltech101, best scratch results were 84.84% at 300 ms for PEV6 with BNext; on N-ImageNet-mini they were 47.86% for EventCount and 43.62% for SAE, both with BNext.

• Representation ranking depends on the dataset, not the architecture: on N-Caltech101 the 6-channel PEV6 is the best encoder for every backbone, reaching 84.84% at 300 ms with BNext and improving up to the full 300 ms extract. On N-ImageNet-mini this inverts: PEV6 collapses, most severely for ResNet-18, which falls to 28.14% (the worst of all), while the dense 2-channel EventCount and SAE become strongest.

• The binary representations saturate earlier: the BEI, which discards event timing and density, peaks at short windows (best 82.42% at 50 ms with BNext, and as early as 20 ms for ReActNet) and then degrades, giving overall the lowest accuracies. From scratch, the best PBEV6 result was 82.54% at 50 ms with BNext, within roughly two points of the PEV6 optimum.

• PBEV6 is competitive at short latency: with BNext (88.45%) or ReActNet (86.79%), PBEV6 surpasses the best full-precision ResNet-18 result (best 86.33% with PEV6) at an 8.6× efficiency gain. Between 10 and 50 ms accumulation times, PBEV6 results are equivalent to SAE and EventCount and closer to PEV6, while allowing binary input layers.

• Cross-modal RGB pretraining helps, especially binary models: pretrained models surpass scratch training for every time window, representation and architecture except A&B BNN with PBEV6. On N-Caltech101, BNext benefits most and most consistently (+5.40 to +6.09), ahead of ReActNet (+3.33 to +4.82), itself ahead of the full-precision ResNet-18 baseline (+2.64 to +3.96). On N-ImageNet-mini, gains are larger for PEV6 (+12.82 for BNext) and PBEV6, with higher relative benefits for ReActNet than BNext.

• A&B BNN has a specific failure mode: pretrained on EventCount it collapses at long accumulation windows on N-Caltech101, falling to 27.63% and below from 100 ms, and on N-ImageNet-mini its EventCount pretrained run failed (recorded as 1.00%). The paper attributes this to its use of scaled weight standardization instead of batch normalization, leaving no activation normalization to absorb EventCount's raw magnitude, which grows with the accumulation window.

• Trained binary signs are uncorrelated with random initialization, but tied to pretrained initialization: from scratch every architecture sits at the 0.500 floor on both datasets. With RGB pretraining, C2I rises to 0.73–0.88. A&B BNN's collapsed N-Caltech101 cells instead reach 0.995, meaning the backbone stayed frozen at its pretrained starting point.

• Complexity gap is large: at 160×160 input resolution, EventBind (ViT-L/14) uses 902.26 M parameters and 10,416,360 M FLOPs; BNext Small uses 67.21 M parameters and 119.57/130.63 M total OPs; ReActNet uses 28.42 M parameters and 51.17/58.54 M total OPs; A&B BNN uses 28.41 M parameters and 49.78/57.15 M total OPs. With binary input, a 6-channel ReActNet costs 47.66 M total OPs versus 51.17 M for its 2-channel full-precision-input version.

• Timing indicators from prior hardware work: the paper does not measure on-device latency. It cites an existing Xilinx ZCU102 FPGA implementation of BNext sustaining 72.6 FPS, estimated at approximately 14 ms per inference, and GCN EFGCN between 4.4 and 9.3 ms per inference on a ZCU104 SoC FPGA, at 62.8 to 64.1% accuracy on N-Caltech101, 20 points below the BNNs at similar 50 ms accumulation windows.

Methodology in Plain English

The authors take three modern deep binary networks (ReActNet, A&B BNN, and BNext Small) plus a full-precision ResNet-18 baseline, and train them on two event datasets: N-Caltech101 (101 classes, 7,000

Authors’ abstract

Binary Neural Networks (BNNs) enable efficient deep learning deployment on resource constrained devices with weights and activations compressed to one bit, substantially reducing model size and inference cost. Event cameras offer complementary advantages, including low latency, high dynamic range, and low power consumption, by capturing asynchronous streams of events rather than dense image frames. Despite their shared emphasis on efficiency, the combination of these technologies remains largely unexplored. This work aims at adapting and evaluating modern deep BNN architectures on event data. We also show that cross-modal pretraining from RGB data can improve the classification accuracy of BNNs on neuromorphic datasets. We introduce the Polar-wise Binary Event Volume (PBEV), a binary representation that enables event-camera data to be processed directly by BNNs and represents a step toward fully binarized event-based vision systems. Best evaluated BNN on N-Caltech101 classification benchmarks shows 90.58% accuracy with 7.5x less operations than their full-precision counterparts.