FMA-Net++: Motion- and Exposure-Aware Joint Video Super-Resolution and Deblurring

AI summary

Overview

Research area: Computer vision, specifically video restoration — the joint task of video super-resolution and deblurring (VSRDB) under dynamically varying camera exposure.

Technical level: Advanced. The paper assumes familiarity with optical flow, transformer attention, dynamic filtering, contrastive representation learning, and standard video restoration benchmarks.

Scope: The paper proposes a non-recurrent, sequence-level architecture that models both motion and per-frame exposure duration to restore sharp high-resolution video from blurry low-resolution input, plus two new exposure-controlled benchmarks.

What This Paper Is About

Blurry low-resolution videos come from two coupled physical causes: the motion in the scene determines the spatial pattern of the blur, and the camera's exposure time determines how much of that motion gets smeared into each frame. Because modern cameras adjust exposure automatically frame by frame, blur severity can change dramatically within a single clip — a situation that nearly all existing restoration methods ignore, since they assume a fixed exposure time. The goal of this paper is a single network that jointly super-resolves and deblurs such videos while explicitly accounting for how long each frame's shutter was open, and that does so efficiently on long sequences.

Key Contributions

1. Hierarchical Refinement with Bidirectional Aggregation (HRBA) blocks. A sequence-level backbone that processes all frames in parallel while hierarchically expanding the temporal receptive field, avoiding both the limited receptive field of sliding-window methods and the sequential bottleneck of recurrent ones.

2. Exposure Time-aware Modulation (ETM) layer. A lightweight modulation layer that conditions HRBA features on per-frame exposure embeddings produced by an Exposure Time-aware Feature Extractor (ETE), which in turn drives exposure-aware Flow-Guided Dynamic Filtering (FGDF) to estimate motion- and exposure-aware degradation kernels.

3. Architectural decoupling of degradation learning from restoration. A Degradation Learning Network (Net D) predicts degradation priors and multi-flow-mask pairs; a Restoration Network (Net R) consumes those priors for efficient high-resolution restoration.

4. Two new benchmarks. REDS-ME (multi-exposure), with five discrete exposure levels corresponding to duty cycles from 1/5 to 5/5, and REDS-RE (random-exposure), which temporally mixes frames across those five levels within REDS4-ME test scenes to create controlled exposure trajectories.

Main Findings

• State-of-the-art accuracy on REDS4-ME at the two hardest exposure levels. At 4/5 and 5/5 exposure, FMA-Net++ reaches 29.66 dB PSNR / 0.8546 SSIM / 1.688 tOF and 29.24 dB / 0.8453 / 1.956 respectively, beating the second-best model (FMA-Net*) by 0.62 dB and 0.73 dB. All comparison methods marked with an asterisk were retrained on the new REDS-ME training set.

• Larger advantage under dynamic exposure. On REDS-RE, where exposure changes within a sequence, FMA-Net++ achieves 30.13 dB PSNR / 0.8643 SSIM / 1.360 tOF versus 29.29 dB / 0.8413 / 1.614 for FMA-Net*, and the margin over other methods widens relative to REDS-ME.

• Strong out-of-distribution generalization. Trained only on synthetic data, FMA-Net++ obtains 30.49 dB PSNR / 0.9018 SSIM / 2.091 tOF on the unseen GoPro dataset, the best of all compared methods.

• Efficiency. FMA-Net++ uses 12.8M parameters and 0.074 s per LR frame at 180×320 resolution, more than a 5.2× speedup over RVRT* (12.9M params, 0.385 s) at higher accuracy. VRT requires 20.5 GB of memory for a 10-frame sequence at 180×320, while FMA-Net++ requires only 6.2 GB.

• HRBA beats alternative temporal designs. On REDS4-ME-5/5, the sliding-window variant scores 28.57 dB / 2.231 tOF at 0.314 s, the recurrent variant 29.11 dB / 1.989 tOF at 0.086 s, and HRBA 29.24 dB / 1.956 tOF at 0.074 s.

• Exposure conditioning helps beyond raw model capacity. A capacity-matched no-ETE variant (13.1M params, slightly larger than the full 12.8M model) absorbs most of the in-distribution PSNR gain on REDS4-ME, but still trails on the exposure-varying REDS-RE split — 29.88 dB PSNR / 1.399 tOF versus 30.13 dB / 1.360 with ETE. The gain on unseen GoPro is reported as +0.64 dB.

• Contrastive ETE outperforms other conditioning designs. On REDS-RE, frame-difference features give 29.75 dB / 1.429 tOF, a classification-pretrained ETE 29.92 / 1.398, an ordinal-regression ETE 29.97 / 1.382, and the contrastive ETE 30.13 / 1.360.

• Guidance is actively used, not ignored. Corrupting only the frame-wise guidance on REDS-RE while keeping input fixed: correct guidance gives 30.13 dB / 1.360 tOF (29.98 / 1.395 on transition frames); same-frame fixed 5/5 guidance gives 29.88 / 1.387 (29.40 / 1.631 on transitions); sequence-shuffled gives 29.59 / 1.525; random wrong-level gives 29.47 / 1.605; no guidance at all gives 29.72 / 1.436. Random wrong-level guidance is worse than no guidance.

• Predicted kernels respond to exposure guidance bidirectionally. For a severely blurred 5/5 input, correct 5/5 guidance yields a spatially diffuse kernel while incorrect 1/5 guidance contracts it; for a mildly blurred 1/5 input, 5/5 guidance makes the kernel more diffuse. t-SNE of ETE features shows clear clustering across exposure levels.

• Not reported in the available content: the paper's data synthesis pipeline details, network configurations, hyperparameter settings, the contrastive loss formulation, and the full Table 8 component ablation are referenced to the Supplementary or cut off by truncation. The content ends mid-sentence in Section 5.3.

Methodology in Plain English

The researchers start from the physics of a blurry frame: a camera integrates the scene over a shutter interval while objects move, then downsamples spatially. Instead of trying to invert that continuous process exactly, they approximate it with a learned, position-dependent degradation kernel that varies across space and time — a kernel that conceptually depends on both the exposure duration and the motion field.

The system has two cooperating networks. Net D looks at the blurry input, predicts optical-flow-plus-occlusion-mask pairs describing motion toward neighboring frames, and predicts per-frame degradation kernels. To make sure these priors are meaningful, Net D is trained by checking that they can actually re-synthesize the original blurry frame from the ground-truth sharp frames. Net R then takes those priors plus the input and reconstructs the sharp high-resolution video, predicting a high-frequency residual that is added to a bilinearly upsampled input.

What connects the two is exposure awareness. A ResNet-18 backbone (the ETE) is pretrained with supervised contrastive learning to separate different exposure durations into distinct clusters in a latent space, then frozen. Each frame's embedding passes through a small network that produces affine scale and shift parameters, which modulate the features inside every refinement block. Because the kernel prediction is derived from features that already carry this exposure information, the dynamic filtering becomes exposure-aware automatically.

Temporal modeling is handled by the HRBA block. Rather than sliding a small window across frames or propagating state frame by frame, HRBA stacks refinement blocks so that each level reaches further into the past and future, aggregating information bidirectionally and in parallel across the whole sequence. Each block also keeps multiple flow-mask hypotheses (n > 1) to handle one-to-many correspondences under severe blur, and uses self-attention plus, in Net R, a cross-attention step where the predicted degradation kernel serves as the query.

Training proceeds in three stages: pretrain the ETE with contrastive loss and freeze it; train Net D with a composite loss combining blurry-frame reconstruction, a warping loss, and optical-flow supervision from a pretrained RAFT (used only during training, not at inference); then train everything end to end.

Why This Matters

Impact on research: The paper reframes VSRDB as a coupling of two physical factors rather than a generic degradation, and shows that exposure duration can be injected as an explicit conditioning signal with negligible overhead. It also argues that the sliding-window versus recurrent trade-off in video restoration is not fundamental, since a hierarchical non-recurrent backbone can beat both while being faster. The two new benchmarks give the field a controlled way to measure robustness to dynamic exposure rather than conflating it with general blur.

Real-world applications:

• Smartphone and consumer camera video, where auto-exposure changes shutter duration frame to frame and produces inconsistent blur within a single clip (the paper demonstrates results on real-world videos captured with a smartphone).
• Surveillance and security footage, where low light forces long exposures and moving subjects smear unpredictably.
• Autonomous driving and robotics perception, where motion-blurred low-resolution frames degrade downstream detection and tracking.
• Archival, broadcast, and user-generated video restoration, where the content was captured with unknown and varying exposure settings and no ground truth exists.

Industry relevance: The efficiency profile matters commercially. At 0.074 s per 180×320 LR frame with 12.8M parameters and 6.2 GB peak memory for a 10-frame sequence, the method is far more deployable on device or in real-time pipelines than transformer alternatives needing 20.5 GB. The fact that it requires only standard RGB input — no event sensors, unlike Ev-DeblurVSR — keeps it applicable to existing camera hardware, and the no-reference quality gains (NIQE, MUSIQ) matter for real footage where reference frames are unavailable.

Future Directions

• Closing the gap to real-world training data. FMA-Net++ is trained solely on synthetic data and evaluated on real videos with no-reference metrics. Learning from unlabeled real footage, or from paired captures with known exposure metadata, is the natural next step.
• Extending exposure awareness to more degradations and tasks. The paper notes that exposure- or event-guided work exists in video deblurring and frame interpolation but does not model motion and exposure jointly in the VSRDB setting. The same ETM/FGDF idea could be applied to denoising, rolling-shutter correction, or high-dynamic-range reconstruction.
• Understanding when the model relies on guidance versus backbone context. The fixed-input sensitivity test shows performance degrades gradually and stays stable even under the farthest wrong guidance, suggesting the HRBA backbone can compensate when strong spatio-temporal evidence exists. Characterizing this fallback behavior, and the failure modes it implies, is an open question.
• Scaling the sequence-level design. The reported memory and runtime figures are at 180×320 resolution with a 10-frame sequence. How HRBA scales to higher resolutions, longer sequences, or streaming/incremental inference is not reported.

Target Audience

This paper is most valuable to computer vision researchers working on video restoration, super-resolution, and deblurring; to PhD students and engineers who need a concrete architectural alternative to sliding-window and recurrent video backbones; and to camera imaging and computational photography engineers who must handle auto-exposure variation in real capture pipelines. Readers should already be comfortable with optical flow, attention mechanisms, and standard restoration metrics (PSNR, SSIM, tOF, NIQE, MUSIQ); those without that background will find the architecture sections dense.

Authors’ abstract

Joint video super-resolution and deblurring (VSRDB) requires both efficient long-range temporal modeling and robustness to frame-wise exposure-duration variation, which changes the extent of motion blur across video frames. We propose FMA-Net++, a non-recurrent, sequence-level framework built from Hierarchical Refinement with Bidirectional Aggregation (HRBA) blocks. By stacking HRBA blocks, FMA-Net++ processes video frames in parallel while hierarchically expanding the temporal receptive field, avoiding the limited temporal receptive field of sliding-window designs and the sequential bottleneck of recurrent ones. To handle exposure-duration-dependent blur, we introduce an Exposure Time-aware Modulation (ETM) layer that conditions HRBA features on exposure embeddings from an Exposure Time-aware Feature Extractor (ETE). The conditioned features guide an exposure-aware flow-guided dynamic filtering module to predict motion- and exposure-aware degradation kernels. FMA-Net++ decouples degradation learning from restoration: the former predicts degradation priors and the latter exploits them for efficient high-resolution restoration. To evaluate VSRDB under controlled exposure-duration variation, we introduce the REDS-ME (multi-exposure) and REDS-RE (random-exposure) benchmarks. Trained solely on synthetic data, FMA-Net++ achieves state-of-the-art accuracy and temporal consistency on these benchmarks. It further shows strong out-of-distribution performance on GoPro and challenging real-world videos, while outperforming recent methods in both restoration quality and inference speed.