Seeing the Unseen: Zooming in the Dark with Event Cameras

AI summary

Overview

• Research area: Computer vision — low-light video super-resolution (LVSR) with event cameras, combining event-based sensing with Retinex-based image decomposition.
• Technical level: Advanced. The paper assumes familiarity with video super-resolution, optical flow alignment, recurrent video architectures, Retinex theory, and event-camera representations (voxel grids).
• One-sentence scope: The paper introduces RetinexEVSR, an event-driven framework that uses illumination and reflectance priors to jointly refine noisy event signals and degraded RGB frames for 4x low-light video super-resolution, and reports state-of-the-art results on SDSD, SDE, and RELED.

What This Paper Is About

Low-light video super-resolution is hard because low-resolution (LR) frames captured in the dark have little contrast and almost no high-frequency detail to recover, so existing methods produce artifacts, structural distortions, and inaccurate illumination. Event cameras can supply complementary high-contrast "moving edge" information even at night, but event data is itself noisy and suffers from trailing artifacts, so naively fusing the two degraded modalities makes things worse. The paper's goal is a framework that extracts useful cues from both degraded signals before fusing them, restoring well-lit high-resolution video.

Key Contributions

1. RetinexEVSR: presented as the first event-driven scheme for LVSR, using high-contrast event signals together with Retinex-inspired priors to restore severely degraded RGB inputs in low light.
2. A Retinex-inspired Bidirectional Fusion (RBF) strategy: illumination guides refinement of noisy event features, and the refined events then recover reflectance details — mutual guidance between RGB and event modalities rather than direct fusion of two degraded signals.
3. Two new modules: an Illumination-guided Event Enhancement (IEE) module that progressively refines event features through multi-scale fusion with illumination, and an Event-guided Reflectance Enhancement (ERE) module that injects high-frequency event information into the reflectance stream via dynamic attention.
4. State-of-the-art results on three datasets (synthetic SDSD and real-world SDE and RELED), with the method reported as robust under extreme darkness and severe motion blur.

Main Findings

• SDSD benchmark gain and speedup: On the SDSD benchmark, the method is reported to achieve up to 2.95 dB gain while reducing runtime by 65% compared to prior event-based methods.
• Per-dataset PSNR improvements over EvTexture: The method improves PSNR by 2.95, 0.95, 0.95, 0.93, and 0.85 dB on five datasets, while reducing FLOPs by 86.1% and runtime by 64.9%, using fewer parameters.
• SDSD and SDE quantitative results (Table 1, 4x LVSR): RetinexEVSR reaches PSNR/SSIM/LPIPS of 30.28 / 0.8932 / 0.3149 on SDSD-indoor, 25.15 / 0.7737 / 0.3933 on SDSD-outdoor, 21.24 / 0.6525 / 0.4627 on SDE-indoor, and 20.68 / 0.6541 / 0.4382 on SDE-outdoor. The strongest event-based baseline reported, EvTexture, gets 27.33 / 0.8776 / 0.3286, 24.20 / 0.7587 / 0.4166, 20.29 / 0.6301 / 0.4869, and 19.75 / 0.6046 / 0.4977 on the same four settings.
• RELED results (Table 2, 4x LVSR): RetinexEVSR reports PSNR 28.92, SSIM 0.8707, LPIPS 0.4612, temporal consistency tOF 4.60 and TCC 3.31 (x10), with 8.07M parameters, 159.1G FLOPs, and 44.5 ms runtime on one 256x320 LR frame. EvTexture reports 28.07 PSNR, 126.9 ms runtime, and 1141.1G FLOPs.
• Generalization to real-world super-resolution (Table 3): Testing the SDE-trained model on SDE without downsampling, the method reports the best or near-best no-reference scores — on SDE-outdoor, NIQE 6.7292, PI 7.0141, CLIP-IQA 0.2618, Q-Align 1.7432, versus EvTexture's 8.0480, 8.4553, 0.2377, and 1.6209.
• Ablation — IEE and ERE both matter (SDSD-indoor): Full model 30.28 / 0.8932 / 0.3149 with 8.07M params; without IEE 28.27 / 0.8642 / 0.3274 (7.26M); without ERE 27.31 / 0.8422 / 0.3304 (6.38M).
• Three scales beat one: Using illumination-guided event features at three scales gives 30.28 PSNR versus 28.64 for a single scale, a reported 1.64 dB improvement; two scales gives 28.83.
• Multi-scale fusion in ERE helps: Single-scale ERE gives 28.04 / 0.8553 / 0.3325 and no-fusion ERE gives 29.78 / 0.8911 / 0.3172, both below the full model.
• Unsupervised Retinex decomposition wins: The unsupervised SCI decomposition reaches 30.28 PSNR versus 29.62 for the larger supervised URetinex-Net, a reported 0.66 dB improvement.
• Flow from reflectance is better than flow from raw frames: Computing optical flow from reflectance gives 30.28 PSNR versus 29.85 from the low-light frames {X_t}, a reported 0.43 dB improvement, attributed to sharper edges for alignment.
• Qualitative behavior: The paper reports that only the proposed method produces well-lit, high-quality results with clearly recognizable text on an extremely low-light (-6.7 EV) example, and that RGB-channel histograms match ground truth more closely than baselines.

Methodology in Plain English

The framework takes a sequence of low-light LR frames plus the corresponding event data and outputs a well-lit high-resolution sequence. Each input frame is first split into an illumination component and a reflectance component using a Retinex-based low-light enhancement model (SCI is used in the final configuration), following the relation X = R ⊙ I.

The key idea is that illumination carries smooth, low-noise global lighting information, while reflectance carries scene content but lacks fine detail in LR video. So instead of fusing the two degraded inputs directly, the network runs a two-way exchange:

• The IEE module uses the illumination map as a global lighting prior to clean up event features. Illumination and event signals go through symmetric lightweight branches, and features are extracted at four spatial scales (full, half, quarter, one-eighth), fused channel-wise with illumination at each scale, then upsampled coarse-to-fine. The top three scales are kept as the enhanced event features. This is meant to suppress trailing artifacts and low-light noise in events.
• The ERE module then takes those refined events and uses them to put high-frequency detail back into the reflectance stream. It has an encoder–bottleneck–decoder structure with attention-based cross-modal fusion at the bottleneck and decoder stages: the query comes from reflectance features, while the key and value come from event features, with dynamic keys/values produced by a Dynamic Contrast Extractor using depth-wise convolutions over a projected temporal dimension D = 4.

Temporally, the model uses a bidirectional recurrent framework in which optical flow acts as a bridge for alignment and feature propagation — but the flow is computed from the reflectance maps, not the raw low-light frames, because reflectance has higher contrast. The enhanced reflectance output also serves as the propagated temporal feature for the next frame.

Finally, event, illumination, and reflectance features are fused to guide upsampling. Training uses 15 input frames, mini-batch 8, 64x64 center crops, random horizontal/vertical flips, 300K iterations with Adam and a cosine annealing schedule, and Charbonnier loss (with ε = 1×10⁻¹²). SpyNet computes optical flow, SCI handles Retinex decomposition, and the SpyNet/SCI learning rate is 2.5×10⁻⁵, frozen for the first 5K iterations, while other modules use 2×10⁻⁴. Training ran on 2 NVIDIA RTX4090 GPUs and took about four days per dataset. Events are voxel grids with 5 temporal bins, downsampled with the same bicubic interpolation as the frames; for SDSD, events are simulated with the vid2e simulator and a noise model based on ESIM.

Why This Matters

Impact on research. The paper argues that combining two degraded modalities directly is the wrong default, and shows a structured alternative: decompose, then use one modality's reliable cue (illumination) to clean the other (events), then use the cleaned modality to enrich the first (reflectance). It also makes a practical case for computing flow from reflectance rather than raw dark frames, and reports that an unsupervised Retinex model outperforms a larger supervised one. The paper positions event-based LVSR as a largely unexplored training problem.

Real-world applications (as named in the paper):

• Video surveillance requiring zoom-in on distant license plates or human faces at night.
• Remote sensing under low-light conditions.
• Night videography.
• Computational photography and surveillance systems generally, where low-light video enhancement is described as critical.

Industry relevance. The reported efficiency profile matters for deployment: the method is stated to reduce FLOPs by 86.1% and runtime by 64.9% relative to EvTexture while using fewer parameters (8.07M in the full model vs 8.90M for EvTexture's reported configuration on RELED), and it runs at 44.5 ms per 256x320 LR frame. That combination of night-time robustness and lower compute is directly relevant to always-on camera pipelines.

Future Directions

• Extending beyond the current degradation model: Main experiments use bicubic degradation, with real-world generalization tested only by running the SDE-trained model on SDE without downsampling — broader real degradation settings remain open.
• Reducing training cost: The paper reports roughly four days per dataset on 2 NVIDIA RTX4090 GPUs, trained from scratch per dataset, which limits how easily the approach scales to more datasets or edge deployment.
• Improving event handling further: The paper shows trailing artifacts are reduced but still frame this as a core difficulty in low light, and explicitly compares against event-only suppression strategies that it deems inadequate — leaving room for better illumination-aware or otherwise guided event cleaning.
• Unifying the pipeline: The approach still depends on external components (a Retinex decomposition model such as SCI and optical flow from SpyNet, with those stages frozen early in training), so learning decomposition and alignment jointly with the rest of the network is a natural next question.

Target Audience

Researchers and graduate students working on video super-resolution, low-light image and video enhancement, and event-based vision; engineers building night-time or low-light camera systems for surveillance, remote sensing, or computational photography; and practitioners interested in multi-modal sensor fusion where both input modalities are degraded rather than clean.

Authors’ abstract

This paper addresses low-light video super-resolution (LVSR), aiming to restore high-resolution videos from low-light, low-resolution (LR) inputs. Existing LVSR methods often struggle to recover fine details due to limited contrast and insufficient high-frequency information. To overcome these challenges, we present RetinexEVSR, the first event-driven LVSR framework that leverages high-contrast event signals and Retinex-inspired priors to enhance video quality under low-light scenarios. Unlike previous approaches that directly fuse degraded signals, RetinexEVSR introduces a novel bidirectional cross-modal fusion strategy to extract and integrate meaningful cues from noisy event data and degraded RGB frames. Specifically, an illumination-guided event enhancement module is designed to progressively refine event features using illumination maps derived from the Retinex model, thereby suppressing low-light artifacts while preserving high-contrast details. Furthermore, we propose an event-guided reflectance enhancement module that utilizes the enhanced event features to dynamically recover reflectance details via a multi-scale fusion mechanism. Experimental results show that our RetinexEVSR achieves state-of-the-art performance on three datasets. Notably, on the SDSD benchmark, our method can get up to 2.95 dB gain while reducing runtime by 65% compared to prior event-based methods. Code: https://github.com/DachunKai/RetinexEVSR.