I2E: Real-Time Image-to-Event Conversion for High-Performance Spiking Neural Networks

AI summary

Overview

Research area: Neuromorphic computing / computer vision — specifically, generating event-stream data for training spiking neural networks (SNNs).

Technical level: Intermediate. The core idea (simulating eye microsaccades with a convolution) is intuitive, but fully appreciating the sim-to-real training paradigm and energy analysis requires familiarity with SNNs and neuromorphic hardware.

Scope: This paper presents an algorithm that converts ordinary static images into synthetic event streams in real time, then uses that data to train SNNs that match or beat models trained on real neuromorphic sensor data.

What This Paper Is About

Spiking neural networks mimic the brain's sparse, event-driven processing, which makes them extremely energy-efficient on neuromorphic hardware — but they need event-stream data (asynchronous pixel-brightness changes captured by special DVS cameras), and that data is scarce, expensive, and slow to collect. Existing datasets are small, often degraded by capture artifacts like screen flicker, and prior software-based conversion methods take over 10 hours just to process ImageNet, ruling out modern training techniques.

I2E solves this by converting static images into high-fidelity event streams in about 0.1 ms per image, fast enough to be embedded directly in a training loop. The goal is to give SNN researchers an unlimited, low-cost, high-quality data source so they can train deep spiking models at scale without waiting on real sensors.

Key Contributions

1. A real-time image-to-event conversion algorithm. I2E runs over 300x faster than prior algorithmic methods and up to 30,000x faster than physical DVS capture, uniquely enabling on-the-fly data augmentation during SNN training — something impossible with static event datasets.

2. Large-scale generated datasets (I2E-ImageNet, I2E-CIFAR100/10). An SNN trained on I2E-ImageNet reaches 60.50% accuracy, a new state-of-the-art for event-based ImageNet that surpasses prior results by more than 8%.

3. A validated sim-to-real training paradigm. Pre-training on synthetic I2E-CIFAR10 and fine-tuning on the real CIFAR10-DVS dataset yields 92.5% accuracy, beating the previous best by 7.7% and showing synthetic events can serve as a high-fidelity proxy for genuine sensor data.

4. Open-source algorithm and datasets. Both the code and all generated datasets are released to accelerate the field.

Main Findings

• Real-time conversion speed: I2E processes one ImageNet-resolution image in roughly 0.1 ms on a GPU, versus 29.47 ms for the previous best method (the ODG algorithm behind ES-ImageNet), a speedup above 300x.

• New ImageNet SOTA for SNNs: An MS-ResNet34 trained on I2E-ImageNet reaches 60.50% accuracy, versus 52.25% for the prior best (ES-ImageNet with ResNet18+LIAF) and roughly 48.93% for N-ImageNet methods.

• On-the-fly augmentation is the single biggest win: Ablation on ImageNet (ResNet18) shows accuracy climbing from 47.22% (naive fixed-threshold conversion) to 48.30% (adding dynamic thresholding) to 49.01% (adding random shift selection) to 57.97% (enabling standard augmentations like random cropping, only possible because conversion is real-time).

• Sim-to-real transfer works: Pre-training on synthetic I2E-CIFAR10 and fine-tuning on real CIFAR10-DVS achieves 92.5%, a 7.7% improvement over the previous state of the art (84.8% with SpikingResformer).

• Massive energy savings: Theoretical analysis on the first ResNet layer shows the I2E-SNN pipeline consumes about 28.68 μJ versus roughly 543 μJ for a conventional ANN layer — an 18.9x reduction, and as much as 75.7x if only 2 timesteps are used.

• Substantial data compression: I2E-ImageNet stored as boolean arrays occupies 47 GB versus 146 GB for the original JPEG ImageNet — a 67.8% reduction. Shannon entropy drops from about 7.14 to 1.53, meaning I2E discards most of the redundant information (uniform textures, backgrounds) while preserving salient features.

• Timestep ordering matters: Frames with higher event rates should be presented first. The optimal order (γ, α, β) yields 89.23% on CIFAR-10, compared to 87.36–88.94% for other orderings.

Methodology in Plain English

I2E mimics what happens when your eye makes tiny involuntary movements (microsaccades) while looking at a scene. Those jitters make the image shift slightly on the retina, which is precisely the kind of change a DVS camera would register.

The pipeline has three stages:

Stage 1 — Intensity map. The RGB image is collapsed into a single channel by taking the maximum of the R, G, and B values at each pixel (the Value channel of HSV). This costs almost nothing computationally and approximates what a sensor's photoreceptors register.

Stage 2 — Simulated motion via convolution. Instead of literally shifting the image eight ways and subtracting pairs (memory-heavy and slow), the researchers express each shift-and-subtract as a single 3x3 convolution kernel containing only one -1 and one +1. Eight such sparse kernels generate eight intensity-change maps in a single parallel GPU operation. During training, one of several equivalent shift vectors is chosen randomly for each direction, adding free data diversity; at inference, a fixed canonical vector is used for deterministic output.

Stage 3 — Adaptive firing. Each pixel fires an ON event if its intensity change exceeds a positive threshold, or an OFF event if it falls below a negative threshold. Rather than a fixed global threshold (which produces wildly different event rates for bright versus dark images), I2E scales the threshold by each image's intensity range, keeping the event rate consistent — about 5% for ImageNet (S_th0 = 0.12) and lower for CIFAR.

Training used MS-ResNet architectures with LIF neurons, the SpikingJelly framework, surrogate gradients (arctan-based) to handle the non-differentiable spike function, and SGD with label smoothing on two RTX 4090 GPUs.

Why This Matters

Impact on research. The paper directly attacks the biggest bottleneck in neuromorphic computing: the lack of large, high-quality event datasets. It shows that synthetic events can substitute for real sensor data in a pre-train-then-fine-tune workflow, which effectively decouples SNN model development from slow, expensive hardware capture. It also unlocks a research direction previously closed off — systematic data augmentation for event streams.

Real-world applications:

• Edge and IoT devices: Any battery-constrained sensor that needs always-on visual recognition (smart cameras, wearables, agricultural monitoring) can benefit from SNNs trained on I2E data while running on neuromorphic chips like Loihi or TrueNorth.
• Robotics and autonomous systems: Event-driven perception offers low-latency motion detection, useful for drones, robotic manipulation, and collision avoidance where power and response time are tight.
• AR/VR and eye-tracking: The microsaccade-inspired encoding is conceptually close to how foveated and gaze-driven vision systems work, which could inform low-power display and tracking pipelines.
• Surveillance and industrial monitoring: Standard RGB cameras — already ubiquitous and cheap — can be turned into effective event-based sensors via a software layer, avoiding the cost of deploying DVS hardware fleets.

Industry relevance. Neuromorphic hardware vendors (Intel, IBM, BrainChip, SynSense) currently lack training data to demonstrate their chips' advantages. I2E provides a scalable data-generation pipeline that lowers the barrier for anyone wanting to build or benchmark spiking models, potentially accelerating commercial adoption of ultra-low-power AI hardware.

Future Directions

• Extending to detection and segmentation. The paper only validates the pre-train/fine-tune paradigm on classification. Whether the same approach transfers to object detection, semantic segmentation, or tracking remains open.

• Systematic study of event-stream augmentation. The paper shows augmentation is a major factor in performance, but the space of valid augmentations for sparse binary event data is largely unexplored.

• Reducing the sim-to-real gap further. The residual 7.5% gap between the 92.5% CIFAR10-DVS result and what presumably real-only training could achieve suggests the synthetic event model (microsaccade simulation) could be made more physically faithful to DVS sensor dynamics.

• Hardware co-design. Since the I2E encoding itself is a lightweight convolution, integrating it directly into neuromorphic sensor front-ends — or into the first layer of an on-chip SNN — could eliminate the conversion step entirely for deployment.

Target Audience

This paper is most valuable to SNN and neuromorphic computing researchers struggling with data scarcity, and to computer vision practitioners interested in energy-efficient alternatives to conventional deep learning. It is also relevant to hardware engineers designing neuromorphic chips who need realistic benchmarks, and to graduate students entering the field who want a practical, well-engineered starting point — the code and datasets are open-sourced, which makes it easy to build on directly.

Authors’ abstract

Spiking neural networks (SNNs) promise highly energy-efficient computing, but their adoption is hindered by a critical scarcity of event-stream data. This work introduces I2E, an algorithmic framework that resolves this bottleneck by converting static images into high-fidelity event streams. By simulating microsaccadic eye movements with a highly parallelized convolution, I2E achieves a conversion speed over 300x faster than prior methods, uniquely enabling on-the-fly data augmentation for SNN training. The framework's effectiveness is demonstrated on large-scale benchmarks. An SNN trained on the generated I2E-ImageNet dataset achieves a state-of-the-art accuracy of 60.50%. Critically, this work establishes a powerful sim-to-real paradigm where pre-training on synthetic I2E data and fine-tuning on the real-world CIFAR10-DVS dataset yields an unprecedented accuracy of 92.5%. This result validates that synthetic event data can serve as a high-fidelity proxy for real sensor data, bridging a long-standing gap in neuromorphic engineering. By providing a scalable solution to the data problem, I2E offers a foundational toolkit for developing high-performance neuromorphic systems. The open-source algorithm and all generated datasets are provided to accelerate research in the field.