EvTexture++: Event-Driven Texture Enhancement for Video Super-Resolution

AI summary

Overview

Research area: Video super-resolution (VSR), multi-modal learning with event cameras, and image/texture restoration.

Technical level: Advanced. The paper assumes familiarity with recurrent VSR backbones, optical flow estimation, deformable alignment, ConvGRU/GRU-based iterative refinement, event-camera data representations (voxel grids), and standard video restoration metrics (PSNR, SSIM, LPIPS).

Scope: A journal extension (IEEE TPAMI, DOI 10.1109/TPAMI.2026.3660020) of a preliminary ICML 2024 paper, presenting an event-driven framework that uses high-frequency event signals for texture restoration rather than only for motion refinement, and packaging it as a plug-and-play module for existing VSR models.

What This Paper Is About

Existing event-based video super-resolution methods use event cameras mainly to improve motion estimation and temporal alignment, but they still over-smooth fine textures and leave visible errors in texture-rich regions. This paper argues that events carry rich high-frequency dynamic detail that is well suited to texture recovery, and builds EvTexture++ to explicitly restore high-resolution textures from events while also using events to keep those textures temporally consistent. The framework is also designed to be attached to other pretrained VSR models (from CNNs to Transformers) to improve their output without retraining the backbone.

Key Contributions

1. A dedicated texture-enhancement paradigm for VSR. EvTexture++ is presented as the first event-driven framework dedicated to texture restoration in VSR, deliberately shifting the role of events from motion refinement to texture enhancement (in contrast to prior event-based VSR work such as EGVSR and EBVSR).

2. An Iterative Texture Enhancement (ITE) module. A specialized texture branch progressively injects high-frequency event information using a voxel-grid event encoding and a shared ConvGRU-based updater that iteratively refines propagated features per voxel bin, instead of injecting events once into the reconstruction process.

3. A Temporal Texture Alignment (TTA) module. This module estimates both event-based flow and RGB-based flow in parallel, producing a texture-aware alignment that reduces temporal inconsistency and texture flickering under large motion.

4. A plug-and-play design. EvTexture++ can be inserted between a frozen pretrained VSR backbone and its upsampler, so only the plug-in and the upsampler are trained, which the authors report yields consistent improvements across different backbones and architectures.

Main Findings

• State-of-the-art 4x results. At 4x upsampling, EvTexture++ reports 26.44/0.8859 (PSNR/SSIM) on the Vid4 "Calendar" clip, 31.82/0.9217 on "City", 28.21/0.8542 on "Foliage", 32.86/0.9381 on "Walk", for a Vid4 average of 29.78/0.8983, plus 32.93/0.9195 on REDS4 and 38.32/0.9558 on Vimeo-90K-T. The preliminary EvTexture reported a Vid4 average of 29.51/0.8909, REDS4 of 32.79/0.9174 and Vimeo-90K-T of 38.23/0.9544.

• Improvement over strong RGB and event baselines. On the Vid4 4x average, IART reaches 28.30/0.8523, MIA-VSR 28.27/0.8519, EBVSR 28.46/0.8701, and EGVSR 24.84/0.7330, all below EvTexture++'s 29.78/0.8983.

• Results at other upsampling scales. At 8x on REDS4 (90x160 to 720x1280), EvTexture++ reports 28.20/0.8053/0.3229 (PSNR/SSIM/LPIPS) versus 28.10/0.7996/0.3288 for EvTexture and 27.14/0.7621/0.3549 for BasicVSR++. At 8x on UDM10 (90x159 to 720x1272) it reports 32.90/0.9046/0.2168. At 2x it reports 45.34/0.9895/0.0414 on Vimeo-90K-T (128x224 to 256x448) and 35.29/0.9668/0.0782 on Vid4 (240x360 to 480x720).

• Plug-in gains up to 1.55 dB. The abstract reports that integrating EvTexture++ into recent VSR models yields gains of up to 1.55 dB in PSNR on the texture-rich Vid4 dataset; Fig. 1 specifically highlights equipping the state-of-the-art RGB-based model IART with the plug-in.

• Qualitative texture recovery. Visual comparisons show frame-based methods (MIA-VSR, IART) and an event-based method (EGVSR) suffering severe over-smoothing on a texture-rich building scene, while EvTexture++ reconstructs coherent building stripes; error maps show lower residuals. A separate figure shows IART alone over-smoothing high-frequency textures and the IART + EvTexture++ plug-in recovering fine structural detail closer to ground truth.

• Motivation for texture focus. The authors state that texture restoration is challenging and underexplored in VSR, and that prior texture work is mostly in single-image super-resolution rather than VSR; to their knowledge they are the first to dedicate a framework to texture restoration in VSR.

• Model/data setting. Event data in the experiments is synthetic: because Vimeo-90K, Vid4, REDS, and UDM10 lack real event data, the authors use the ESIM event simulator to generate events from video frames.

Methodology in Plain English

The framework is built on the recurrent VSR backbone BasicVSR, processing an LR frame sequence together with the events occurring between frames. At each timestep it runs two parallel branches, both guided by events:

• Texture branch. Events between two frames are turned into a voxel grid with B = 5 time bins and normalized using the 98th percentile of non-zero values to suppress hot pixels. A context extractor (eight residual blocks, as used in ESRGAN) encodes the current LR frame, while a custom five-layer U-Net encodes each voxel bin separately. A shared texture updater of three ConvGRU layers and five residual blocks then walks through the voxel bins one at a time, adding a learned residual at each iteration (the number of iterations equals the number of voxel bins). The final texture-enhanced feature is the input feature plus the sum of all per-bin residuals, so the refinement is residual rather than a direct network output.

• Motion branch and TTA. A Temporal Texture Alignment module runs two aligned streams: one warps the previous feature using flow estimated from voxelized events via a U-Net (following event-based frame interpolation design), and one warps it using RGB optical flow from the lightweight SpyNet. The two warped features are concatenated and fused with a 1x1 convolution. The idea is that events capture fast, non-linear motion while RGB flow gives coarse but reliable appearance cues.

• Fusion and output. The texture-enhanced and motion-enhanced features are combined in a fusion module of fifteen residual blocks, along with the backward-propagated feature and current LR frame for context. The fused feature is upsampled by a pixel-shuffle layer and added to a bicubic-upsampled LR frame. Propagation is bidirectional, with forward and backward paths interconnected.

• Plug-and-play variant. When used as a plug-in, the pretrained backbone stays frozen and emits three intermediate outputs: pre-propagation spatial features, post-propagation temporal features, and bidirectional optical flows. The plug-in refines the post-propagation feature using events and the other backbone outputs. The internal optical flow estimator is not jointly trained with the plug-in (so training stays cheap), and the plug-in reuses the backbone's flow instead of running its own SpyNet. This is motivated by the cost of training Transformer-based VSR models: the paper cites roughly 27 days on 8 RTX 4090 GPUs for IART on REDS and about 45 days for MIA-VSR, based on their original papers.

Why This Matters

The work matters because it repositions event cameras in video restoration: rather than treating events as a motion aid, it treats them as a source of high-frequency texture detail, and it shows that the resulting module can be bolted onto already-trained, expensive-to-train VSR backbones with minimal computational overhead. Event cameras are described as offering ultra-high temporal resolution (around 1 microsecond), high dynamic range (120 dB), and low power consumption (approximately 5 mW), which makes this direction relevant wherever frame-based capture struggles.

Real-world applications discussed or implied by the paper:

• High-definition television and broadcast upscaling, where fine texture detail and stable playback matter.
• Video surveillance, where detail recovery and wide dynamic range conditions are important.
• Virtual reality, where visual fidelity of reconstructed video affects the experience.
• General video enhancement, including low-resolution footage restoration.

Industry relevance: one co-author is affiliated with Midea Group, and the plug-in formulation directly targets the practical cost of training large Transformer-based VSR models (cited as week-scale runs on multiple high-end GPUs), allowing organizations to improve existing deployed backbones rather than retrain them. The low-power event sensor profile is also relevant to embedded and mobile capture pipelines.

Future Directions

• Closing the simulation gap. The experiments rely on events generated by the ESIM simulator because the standard benchmarks lack real event data; validating EvTexture++ on real event-camera captures, and characterizing how simulated-to-real differences affect texture restoration, remains open.

• Generalization across more backbones and scales. The paper tests the plug-in on several architectures including IART (and mentions PSRT and MIA-VSR in the plug-in design), but the full space of VSR backbones, and behavior under other degradations beyond the reported blur-downsampling (BD) robustness analysis, is not settled.

• Handling asynchronous event alignment more robustly. The authors explicitly frame the asynchronicity of events versus synchronous RGB frames as a risk of spatiotemporal misalignment that could introduce artifacts; improving or self-supervising this alignment is a natural next step.

• Extending texture restoration to joint space-time super-resolution. The related work notes existing event-based space-time VSR efforts (for example EvSTVSR for large-motion scenarios); combining explicit event-driven texture recovery with temporal upsampling is an open direction.

Target Audience

Researchers and practitioners working on video super-resolution, event-based vision, multi-modal video restoration, and efficient model reuse or plug-in adaptation of large pretrained restoration models. It is also relevant to engineers deploying super-resolution on GPU-constrained or mobile hardware who want measurable quality gains from a small add-on module, and to readers interested in how event sensors can contribute beyond motion estimation. Readers without background in recurrent VSR, optical flow, or event representations will find the equations and terminology demanding.

Authors’ abstract

Event-based vision has drawn increasing attention owing to its distinctive properties, including ultra-high temporal resolution and extreme dynamic range. Recent works have introduced it to video super-resolution (VSR) to enhance flow estimation and temporal alignment. In contrast, this paper shifts the focus of event signals from motion refinement to texture enhancement in VSR. We propose EvTexture++, the first event-driven framework dedicated to texture enhancement in VSR. It leverages high-frequency spatiotemporal details from events to improve texture recovery. EvTexture++ incorporates a customized texture enhancement branch, along with an iterative texture enhancement module that progressively exploits high-temporal-resolution event information for texture restoration. This enables gradual refinement of texture regions across iterations, yielding more accurate and detailed high-resolution outputs. Besides intra-frame texture recovery, large motions could degrade inter-frame temporal consistency, particularly in texture regions, leading to texture flickering. To mitigate this, we further exploit the continuous-time motion cues of events to enhance temporal consistency, introducing a temporal texture alignment module that estimates event-guided texture-aware flow for precise inter-frame texture alignment. Moreover, EvTexture++ is designed as a plug-and-play tool to flexibly boost the performance of existing VSR models. Experiments on five datasets demonstrate that EvTexture++ achieves state-of-the-art performance. When integrated into recent VSR models, it yields significant improvements, with gains of up to 1.55 dB in PSNR on the texture-rich Vid4 dataset. Code: https://github.com/DachunKai/EvTexture.