EvDiff: Event-Based Video Reconstruction using One-Step Diffusion Models

AI summary

Overview

Research area: Computer vision — event-based (neuromorphic) sensing and generative video reconstruction, combining event cameras with large-scale one-step diffusion models.

Technical level: Advanced. The paper assumes familiarity with diffusion models, latent variable models (VAEs), transformer backbones (DiT), and event-camera representations such as voxel grids.

Scope: This paper proposes EvDiff, a one-step diffusion framework built on Stable Diffusion 3 that reconstructs high-frame-rate, chromatic video from monochromatic event streams, trained without relying on large paired event–image datasets.

What This Paper Is About

Event cameras record only per-pixel brightness changes as sparse, asynchronous events, never absolute intensity. Recovering ordinary video frames from those events is therefore severely ill-posed. Existing regression-based methods (the E2VID family) map events directly to frames using relatively small paired datasets and compact networks, producing low-quality grayscale output with blur, ghosting, and block-like artifacts. This paper's goal is to bring large pretrained generative priors into this task while keeping inference fast enough for high-frame-rate video.

Key Contributions

1. Diffusion priors for event reconstruction. The paper reformulates event-to-video reconstruction as two stages — a degraded intermediate frame reconstruction followed by a high-capacity generative refinement — and instantiates the second stage with a one-step diffusion model using Stable Diffusion 3 as the base model.

2. A Surrogate Training Pipeline. A three-stage training recipe (DiT training, surrogate distillation, joint fine-tuning) that replaces the need for large-scale event–image paired data with widely available large-scale image datasets, using the paper's Places365 training of 1.8 million (1,800,000) images.

3. An E2VID-style Degradation Model. A synthetic degradation pipeline that models three identified sources of error in regression-based event reconstruction — the event generation mechanism (EGM), sensor-inherent imperfections (SII), and algorithmic degradation (AD) — so that high-quality images can be converted into realistic low-quality stand-ins.

4. EvEncoder with Efficient Temporal Fusion (ETF). A recurrent event encoder with a lightweight gated fusion module that reduces the complexity of the multi-frame setting from O((THW)²·C) to O((HW)²·T·C), enabling efficient high-frame-rate processing.

Main Findings

• Best reported MSE, LPIPS and FID on both evaluation datasets. On BS-ERGB, EvDiff reports MSE 0.0463, SSIM 0.3394, LPIPS 0.4023, FID 148, FVD 984. On DSEC, it reports MSE 0.0476, SSIM 0.3677, LPIPS 0.4226, FID 129, FVD 1491. The paper states these correspond to improvements of 32.2%/8.5%/37.8% (MSE/LPIPS/FID) over the second-best method on BS-ERGB and 20.8%/14.0%/44.9% on DSEC.

• SSIM and DSEC FVD are not led by EvDiff. On BS-ERGB the highest SSIM in Table 1 belongs to E2VID+ (0.3710) versus EvDiff's 0.3394; on DSEC the highest SSIM is SPADE-E2VID (0.4578) versus EvDiff's 0.3677. The lowest FVD on DSEC is SPADE-E2VID (1277) versus EvDiff's 1491.

• Only method producing color from monochrome events. The paper states EvDiff is the only compared method that generates chromatic video from monochromatic event streams, and that its reconstructions can show higher dynamic range than the ground truth (for example, retaining cloud shape where the GT frame is over-exposed).

• Lower cost than a ControlNet-based counterpart. Table 2 reports 2.19G parameters for EvDiff versus 8.23G for the ControlNet baseline, and 2.18T FLOPs at 512×512 versus 26.62T / 49.93T / 96.56T for ControlNet at 10 / 20 / 40 steps; at 1024×1024 the figures are 9.56T versus 87.15T / 163.55T / 316.36T.

• ControlNet needs prompt information that is unavailable in practice. With scene-agnostic "Fixed Prompt" inputs, the paper reports that the ControlNet-based approach largely fails to reconstruct, whereas with prompts derived from ground-truth images it is still less temporally consistent and less faithful than EvDiff.

• Each training stage matters. Ablations report a 56.8% performance drop when Stage 1 is removed, an 83.6% decrease without the E2VID-Style Degradation Model, and a 29.8% gain from Surrogate Distillation. Removing Stage 3 joint fine-tuning also degrades results.

• Degradation components all contribute. Excluding any one of EGM, SII, or AD leaves performance below the full model on both BS-ERGB and DSEC.

• ETF outperforms standard recurrent alternatives. The paper reports ETF improves MSE and FVD by 20.9% and 24.2% respectively, and that replacing it with ConvLSTM or ConvGRU (at comparable model size) consistently degrades performance. Replacing EvEncoder with an "E2VID + VAE-Encoder" baseline also degrades perceptual quality while making the pipeline 2.5× larger and slower (84 ms vs. 63 ms).

• Two stated limitations. The framework produces residual temporal flicker because temporal consistency is encouraged mainly through frame-wise latent consistency, and the SD3 backbone occasionally yields faint periodic grid-like artifacts attributed to the patch-based DiT architecture.

Methodology in Plain English

The authors split the reconstruction problem in two. First, a degraded frame is produced from events; second, a powerful generative model cleans that degraded frame up. This split lets the generative half be trained on ordinary photographs, because the authors observed that regression-based event reconstructors fail in fairly consistent ways.

To exploit that, they built a synthetic degradation model. Starting from a high-quality image, it simulates the three error sources they identified: the ambiguity of absolute brightness (creating blotchy, uneven texture), sensor imperfections such as trailing events and noise (creating edge damage and grain), and algorithmic losses from voxel accumulation and limited reconstruction capacity (approximated with motion blur and down-then-up resampling).

Training then proceeds in three stages. Stage 1 trains a Diffusion Transformer and a trainable "surrogate" VAE encoder on pairs of clean and synthetically degraded images from Places365 (1.8 million images), using a loss combining latent reconstruction and LPIPS. Stage 2 replaces the surrogate VAE encoder with the event-specific EvEncoder and distills it to match the VAE latent space, using an off-the-shelf E2VID model to turn events into the degraded form during training. Stage 3 jointly fine-tunes everything on a small real event–RGB dataset with perceptual, flow-based temporal, and latent consistency losses. At inference the model is end-to-end and single-step: events go in, a latent comes out of a one-step diffusion update, and the VAE decoder produces the frame. All models are fine-tuned with LoRA at rank 64, optimized with AdamW at an initial learning rate of 5×10⁻⁶ (β₁ = 0.9, β₂ = 0.999) on a single NVIDIA H200 GPU, with 180k iterations for Stage 1 (batch size 10), 12k iterations for Stage 2 (batch size 1, sequence length 40), and 12k iterations for Stage 3 (batch size 1, sequence length 30).

Why This Matters

Impact on research. The paper offers a concrete route around the data bottleneck that has constrained event-based vision: instead of collecting orders of magnitude more paired event–image data (which the authors argue is nearly impossible because event cameras are not widely adopted), it substitutes a synthetic degradation model and large unpaired image datasets. It also demonstrates that one-step diffusion, previously shown useful in image restoration tasks, transfers to a high-frame-rate video sensing modality.

Real-world applications:

• High-speed and high-dynamic-range video capture where conventional cameras blur or clip, such as fast motion or scenes with extreme lighting.
• Low-light and nighttime imaging, building on event cameras' ability to sense changes where frame-based sensors fail.
• Robotics and autonomous navigation, where event-based pipelines are used for SLAM, feature tracking, and robustness under challenging illumination.
• Video frame interpolation and deblurring workflows, where reconstructed intensity frames from events can supplement conventional footage.

Industry relevance. The efficiency comparison is the commercially salient part: 2.19G parameters and 2.18T FLOPs at 512×512 with a single forward step, versus 8.23G parameters and 26.62T FLOPs for the 10-step ControlNet baseline, with linear rather than quadratic scaling in frame count. That gap matters for deployment on latency-sensitive hardware, and the paper notes the code will be released publicly.

Future Directions

• Reducing residual temporal flicker. The authors identify frame-wise diffusion with only latent-level temporal consistency as the cause of slight illumination and appearance flicker in dynamic scenes; stronger temporal modeling is an open problem.
• Removing backbone artifacts. The faint periodic grid-like artifacts traced to the patch-based DiT architecture of SD3 suggest that alternative backbones or architectural adjustments are worth investigating.
• Replacing the synthetic degradation surrogate. Since the E2VID-style degradation model is a hand-designed approximation of real reconstruction errors, learning or validating it against a wider range of real event data could improve transfer.
• Broadening evaluation. The paper excludes HQF and MVSEC for quality and color reasons, and evaluates on BS-ERGB and DSEC only; wider real-world benchmarking, including metrics regimes where SSIM or DSEC FVD favor other methods, remains open.

Target Audience

Researchers and graduate students working on event-based vision, neuromorphic sensing, and diffusion-based image or video restoration. It is also relevant to practitioners building low-latency vision systems (robotics, automotive, HDR and high-speed imaging) who need to weigh generative quality against inference cost. Readers without background in diffusion models or event-camera event representations will find the method sections demanding.

Authors’ abstract

As neuromorphic sensors, event cameras asynchronously record changes in brightness as streams of sparse events with the advantages of high temporal resolution and high dynamic range. Reconstructing intensity images from events is a highly ill-posed task due to the inherent ambiguity of absolute brightness. Early methods generally follow an end-to-end regression paradigm, directly mapping events to intensity frames in a deterministic manner. While effective to some extent, these approaches often yield perceptually inferior results and struggle to scale up in model capacity and training data. In this work, we propose EvDiff, an event-based diffusion model that follows a surrogate training framework to produce high-quality videos. To reduce the high computational cost of high-frame-rate video generation, we design an event-based diffusion model that performs only a single forward diffusion step, equipped with a temporally consistent EvEncoder. Furthermore, our novel Surrogate Training Framework eliminates the dependence on paired event-image datasets, allowing the model to leverage large-scale image datasets for higher capacity. The proposed EvDiff is capable of generating high-quality colorful videos solely from monochromatic event streams. Experiments on real-world datasets demonstrate that our method strikes a sweet spot between fidelity and realism, outperforming existing approaches on both pixel-level and perceptual metrics. The code will be released publicly.