EvReflection: Event-Driven Micro-Dynamics for Reflection Removal

AI summary

Overview

Research area: Computer vision, specifically reflection removal from images; at the intersection of event-based vision and cross-modal image restoration.

Technical level: Advanced. The paper combines event-camera signal modeling, a theoretical solvability proof built on the optical flow constraint, and a two-stage deep architecture (FocalNet context encoder, recurrent disentanglement, cross-attention rectification).

Scope in one sentence: This paper proposes EvReflection, an event-camera-driven network that exploits micro-dynamics from subtle handheld motion to separate reflection and transmission layers, together with a parallax-aware simulation pipeline and the real-world EVR2 benchmark dataset.

What This Paper Is About

Reflections on glass, such as a window, mix with the background the camera is trying to capture, and separating the two from a single photo is mathematically ambiguous because infinitely many transmission/reflection pairs can produce the same observed image. The authors' goal is to break that ambiguity by adding an event camera, which records brightness changes at microsecond resolution, so that the tiny differential motion between the reflection (on the glass surface) and the background (behind it) becomes usable as a physically grounded separation cue. They build the network to exploit this cue, plus a simulation pipeline and a real-world dataset to train and test it.

Key Contributions

1. A new perspective for reflection removal: introducing event signals to exploit micro-dynamics induced by subtle camera motion, rather than requiring the large camera baseline that multi-frame RGB methods depend on.
2. The EvReflection architecture, combining a Micro-Dynamics Decoupler (MDD) that disentangles layer-specific motions from event streams into texture-agnostic dynamic priors, and a Parallax-Attention Rectifier (PAR) that uses those priors to spatially modulate RGB features for artifact removal.
3. A parallax-aware simulation pipeline that models optical geometry and 3D camera trajectories to synthesize physically consistent RGB-event training data, addressing paired-data scarcity.
4. The EVR2 benchmark dataset, described as the first real-world dataset for this task, captured with a synchronized Event-RGB hybrid camera on a motorized linear slide across three glass thicknesses.

Main Findings

• Synthetic benchmark (SIR2): EvReflection (labeled S+E, meaning it uses events in addition to a single image) reaches an average of 29.09 dB PSNR, 0.940 SSIM, and 0.036 LPIPS across the Objects (200), Postcards (199), and Wild (55) subsets. The best competing method, DAI, averages 27.46 dB PSNR, 0.919 SSIM, 0.092 LPIPS.
• Margin of improvement: The abstract states EvReflection surpasses the best competing method by more than 1.6 dB and 1.2 dB in PSNR on the synthetic and real-world benchmarks, respectively.
• Real-world benchmark (EVR2): EvReflection averages 27.25 dB PSNR, 0.896 SSIM, 0.063 LPIPS across the T3, T5, and T8 subsets, versus 25.97 dB PSNR for RDNet, the next best method. Per-subset results are 28.88 dB (T3), 27.04 dB (T5), and 25.81 dB (T8).
• Performance degrades with glass thickness: All methods drop from T3 to T8. The authors attribute this to thicker glass inducing larger spatial offsets between front- and back-surface reflections, creating double-reflection ghosting. DAI, which is strongest on SIR2, drops to 21.25 dB average on EVR2 when evaluated with official pre-trained models.
• Trained on less data yet generalizes better: The model is trained only on 7,643 synthetic image pairs from PASCAL VOC, while most baselines use larger training sets that often include real-world data such as Real and Nature. For real-world results, the authors use a hybrid strategy mixing the synthetic VOC data with 387 real-world samples from EVR2.
• Event modality is essential (ablation): An RGB-only baseline scores 26.26 dB PSNR / 0.885 SSIM on EVR2. Adding events without dynamics decoupling gives 26.74 dB / 0.887.
• Implicit dynamics beat explicit optical flow (ablation): Using explicit optical flow as the dynamics cue reaches 27.09 dB PSNR / 0.896 SSIM, with SSIM matching the full model but PSNR lower because flow estimation is unstable in texture-less or repetitive regions. The full MDD model reaches 27.25 dB / 0.896.
• Attention-based fusion beats concatenation (ablation): Replacing PAR with simple feature concatenation drops to 27.02 dB / 0.889, versus 27.25 dB / 0.896 for the full model.
• Generic event-based restoration networks underperform: Retrained on EVR2, EvLight (low-light enhancement) reaches 23.46 dB PSNR, DeblurSR (super-resolution) 23.61 dB, and EFNet (motion deblurring) 24.24 dB, all well below EvReflection's 27.25 dB. The authors attribute this to those methods assuming a single degradation model rather than needing to decouple two additive layers.
• Theoretical solvability condition: The closed-form solution to the derived linear system depends on the Hessian becoming positive definite, which holds when the motion vectors of the two layers differ (v_T ≠ v_R). The authors state this means distinct micro-dynamics from subtle camera shake guarantee a unique solution.
• Motion-bandwidth paradox for frame-based methods (appendix): With minimal relative motion, there is no parallax to separate layers; increasing velocity to induce parallax increases blur length, narrowing the Sinc filter passband and irreversibly attenuating the high frequencies needed for edge-based separation.

Methodology in Plain English

The approach rests on a physical observation: if the camera behind glass moves even slightly, the reflection on the glass surface and the scene behind it shift by different amounts, because they sit at different depths. The authors first write down the equations for how an event camera responds to the scene and show that, when the two layers move differently, the layer separation problem becomes solvable. An event camera is used because it detects brightness changes with microsecond precision, so it catches these tiny differential displacements that normal video frames blur over.

The network takes a mixed RGB image and the concurrent raw event stream. Events are first converted into a spatiotemporal voxel grid to preserve microsecond motion detail. A Micro-Dynamics Decoupler then processes this grid: an event encoder with five residual blocks produces an initial dynamics embedding, and a stack of dual-branch Dynamics Disentangle Blocks maintains two evolving states, one for transmission dynamics and one for reflection dynamics, initialized identically and updated while looking at each other's previous state. Low-level RGB features are injected at each iteration to provide high-frequency spatial boundaries that anchor the motion estimation. The two decoupled streams are concatenated into a single dynamic prior.

A Parallax-Attention Rectifier then pairs that prior with RGB content. A pre-trained FocalNet extracts hierarchical semantic features; the shallow features anchor the decoupler and the deep features serve as the reconstruction substrate. Both the deep visual features and the dynamics prior pass through self-attention blocks, then are projected into query (from RGB) and key (from dynamics) to compute a spatial attention map via softmax similarity. High attention scores mark regions where visual texture agrees with transmission micro-dynamics, and low scores mark reflection-dominated regions. The prior is also projected into a value embedding, multiplied element-wise by the attention map, and added residually to the deep RGB features. A feed-forward network and decoder produce the final clean transmission estimate.

For data, since capturing pixel-perfect ground truth with synchronized events is impractical, the authors built a simulation pipeline. They take images from PASCAL VOC and SIR2, assign different virtual depths to transmission and reflection layers, move a virtual camera along a smooth random 3D trajectory, and compute separate homography matrices per layer to warp each into a five-frame sequence. Layers are blended with the DSRNet physical model that includes possible non-linearities. Because direct low-frame-rate-to-event conversion causes temporal aliasing, the mixed video is first interpolated to a high frame rate and then converted through the ESIM simulator into asynchronous events.

For the real dataset, an Event-RGB hybrid camera at 608 × 768 is mounted on a motorized linear slide that produces smooth, repeatable horizontal translation. They recorded 140 distinct scenes through glass of 3 mm, 5 mm, and 8 mm thickness, giving 420 paired sequences, and used performance-guided stratified sampling to pick 11 representative test scenes spanning faint translucent reflections to heavy saturated interference.

Training used PyTorch on 8 NVIDIA RTX 4090 GPUs, batch size 2 per GPU for a total of 16, Adam with β1 = 0.9 and β2 = 0.999 and no weight decay, an initial learning rate of 1 × 10⁻⁴ with cosine annealing over 55 epochs, and random cropping plus horizontal flipping. The loss follows RDNet's objective: a weighted combination of reconstruction loss (MSE plus gradient terms) and VGG-based perceptual loss.

Why This Matters

Impact on research: The paper offers a theoretically motivated reason why event cameras help with a problem that RGB-based reflection removal has treated as inherently ill-posed, and it contributes both a simulation pipeline and the first real-world benchmark for the task. That combination gives the subfield a reproducible evaluation target and a data-generation route that avoids the near-impossible task of capturing simultaneous pixel-perfect ground-truth layers and high-speed events in the wild.

Real-world applications:

• Photography and mobile imaging through windows, glass display cases, or shopfronts, where users want the scene behind the glass rather than the reflection of what is behind the camera.
• Surveillance and security cameras mounted behind glass, where reflections of the room or street can obscure the monitored scene.
• Robotics and autonomous navigation, where perception through transparent surfaces (windows, glass doors, protective panels) must not be corrupted by reflected content.
• Object detection and downstream vision pipelines, which the authors note are hindered by reflection-degraded inputs.

Industry relevance: The method is designed for casual handheld capture with only subtle motion, which is the realistic operating regime for consumer devices, unlike multi-frame RGB approaches that need a large camera baseline. Event sensors are already shipping in hybrid camera products (the paper uses a Shimeta Lingguang-1 with an AlpsenTek ALPIX-Eiger sensor), so the approach is plausible for integration where synchronized RGB and event streams already exist. Counterweights are the reported computational overhead of the large backbone and the failure modes in static tripod setups and extreme low light.

Future Directions

• Motion-free priors: The authors state EvReflection struggles in strictly static scenes such as tripod setups, where minimal motion yields no useful event signal, and plan to incorporate motion-free priors to cover this case.
• Robustness in extreme low light: Severe sensor noise at low illumination degrades both the RGB and event modalities, a limitation the authors flag but do not resolve.
• Lightweight architectures: The large backbone incurs substantial computational overhead, and the authors list developing lightweight alternatives as future work.
• Extending the event-guided separation idea: The conclusion positions this as a foundation for event-guided image separation tasks more broadly, suggesting generalization to other layer-decomposition problems as an open direction. The authors also note they are the first to introduce event cameras for reflection removal, so there are no direct competitors yet and the design space is largely unexplored.

Target Audience

Researchers and graduate students in computer vision working on reflection removal, computational photography, or image restoration; event-camera and neuromorphic-vision researchers looking for new low-level vision applications; and engineers building perception stacks for robotics, surveillance, or mobile imaging who need to handle reflective transparent media. Readers need comfort with event-camera signal models, optical flow constraints, attention mechanisms, and standard restoration metrics (PSNR, SSIM, LPIPS) to follow the derivations and comparisons.

Authors’ abstract

Despite remarkable progress in reflection removal, current methods primarily exploit static image priors from a single frame and still suffer from severe residual artifacts due to the inherent ambiguity between the reflection and transmission layers. In this paper, we propose leveraging event signals to break this ambiguity. By employing event cameras to capture micro-dynamics, we reveal the differential motion between these two layers. We thereby present a novel event-driven reflection removal network, EvReflection, that utilizes these dynamic cues for layer separation. Specifically, we design a Micro-Dynamics Decoupler to disentangle layer-specific motions from event streams as priors, which then guide a Parallax-Attention Rectifier to cleanly remove artifacts from the RGB image. Furthermore, to address data scarcity, we develop a parallax-aware simulation pipeline and construct the EVR$^2$ benchmark dataset, the first real-world dataset for this task. Extensive experiments demonstrate that EvReflection achieves state-of-the-art performance on both synthetic and real-world benchmarks, surpassing the best competing method by more than 1.6 dB and 1.2 dB in PSNR, respectively. The code, dataset, and pre-trained models are available at https://github.com/JiaxiaoWang/EvReflection.