Modulo Video Recovery via Selective Spatiotemporal Vision Transformer

AI summary

Overview

Research area: Computational imaging and video restoration — specifically recovering high-dynamic-range (HDR) video from modulo-sampled (folded) sensor data, using a Transformer-based deep learning model.

Technical level: Advanced. The paper assumes familiarity with Vision Transformers, self-attention, HDR tone mapping, and modulo arithmetic.

Scope: The paper proposes SSViT (Selective Spatiotemporal Vision Transformer), the first deep learning framework for modulo video reconstruction, and evaluates it on two HDR video datasets against four baselines.

What This Paper Is About

Ordinary camera sensors have limited dynamic range, so very bright regions of a scene saturate and lose information. A modulo camera avoids saturation by letting each pixel counter wrap around to zero when it reaches its maximum, producing samples that are the true brightness values modulo 2^A — but these wrapped ("folded") samples must be mathematically unwrapped to recover the real signal. This paper tackles the harder video version of that unwrapping problem: given a clip of 8-bit folded frames, recover the underlying higher-bit-depth HDR video by predicting, for every pixel, how many times it was folded.

Key Contributions

1. Demonstrates that standard HDR and standard video methods fail at modulo recovery. The authors argue both analytically and empirically that traditional HDR reconstruction (which assumes continuous frame dependencies) cannot estimate the discrete folding numbers, and that existing video attention models (e.g., TDNet) only look at the nearest one to three frames.
2. Introduces SSViT, the first deep learning framework for modulo video reconstruction. It adapts existing Transformer modules — spatiotemporal attention and a modified ViT-style encoder — rather than designing an architecture from scratch, and removes positional embeddings because token positions are deemed not critical for this task.
3. Proposes a token selection strategy that splits each frame's embedded features into "basic" areas (consistent folding labels) and "intricate" areas (variable folding labels), spending computation only on the intricate ones. Selection uses an extended 3D Neighboring Similarity Matrix (NSM) combining KL divergence and cosine distance.
4. Validates the method experimentally, reporting reconstruction of 12-bit videos from 8-bit modulo inputs and outscoring Uformer, EPCE, the MRF-based method, and UnModNet on the LiU and HdM datasets.

Main Findings

• SSViT leads on the LiU dataset. PSNR 28.85 dB and SSIM 0.871, versus UnModNet at 27.71 dB / 0.811, EPCE at 13.59 dB / 0.597, MRF at 12.43 dB / 0.493, and Uformer at 11.38 dB / 0.482.
• SSViT also leads on the HdM dataset. PSNR 29.38 dB and SSIM 0.850, versus UnModNet at 28.03 dB / 0.824, EPCE at 16.03 dB / 0.589, MRF at 15.84 dB / 0.577, and Uformer at 14.36 dB / 0.535.
• Traditional HDR methods are far behind. Uformer and EPCE — which take LDR frames rather than modulo frames as input — score roughly 11–16 dB PSNR, while the modulo-specific methods (UnModNet, SSViT) score above 27 dB, which the authors present as evidence for their claim that HDR recovery is unsuitable for modulo recovery.
• HDR-style methods fail for a different reason than modulo methods. EPCE does not produce incorrectly recovered modulo pixels, but it loses brightness information because saturated pixels are sampled at maximum value in its LDR input, hurting recovery in extremely bright regions.
• All algorithms struggle on disordered or abstract patterns in the modulo frame, which the authors describe as a limitation of such methods generally.
• Qualitative gain example. In the sequence labeled Bridge.mp4, SSViT reportedly reconstructs the water and road surface more accurately than the other methods, which the authors attribute to its use of long-range dependencies to estimate folding numbers.
• Tone mapping matters for video. Because most tone mapping operators scale HDR luminance per frame independently, flicker appears; the authors compute each frame's average brightness and adjust it against the previous frame, plus apply Reinhard tone mapping using the global average brightness of the whole video.
• Not directly comparable baseline setting. The authors note UnModNet, MRF, Uformer, and EPCE are not designed for video and process individual frames; they also state the comparison "may not be entirely equitable" because Uformer and EPCE belong to the conventional HDR reconstruction category.

Methodology in Plain English

The core insight is that unwrapping a modulo image is really a per-pixel classification problem: for each pixel, decide how many times the value wrapped around (its "folding number"). Pixels sharing the same folding number can be scattered all over a frame, so the model needs to compare distant pixels — which is why the authors chose a Transformer.

To make this tractable, they reformulate folding as an iterative binary prediction: instead of guessing a large folding number directly, at each round the network predicts a binary mask saying "is this pixel folded at least one more time?" The mask is added back into the modulo frames, and the process repeats until no pixel is predicted folded again. The inference loop runs while k < 2^(B−A).

The network has four parts: a shared encoder that embeds all frames in a clip, a token selection module, a spatiotemporal Transformer backbone, and a decoder that predicts the folding mask. Each clip is encoded once and features are reused, allowing parallel reconstruction of frames.

To cut cost and focus on hard regions, the token selection module computes an NSM score per location using a neighborhood radius r (the neighborhood has (2r+1)³ elements). High NSM scores mark intricate regions — sharp fold edges and abrupt feature changes — and only those tokens are processed by the spatiotemporal Transformer. Unselected tokens get their folding mask via optical flow: FlowNet2 estimates motion between frames and a warping operation pulls the mask from a historical frame. This hybrid is meant to preserve temporal consistency cheaply.

Training details: PyTorch, 200,000 iterations, one NVIDIA GeForce RTX 4090 GPU with a 12th Gen Intel Core i7-12700KF processor, Adam optimizer at an initial learning rate of 1e-4, clip length n_c = 4, cross-entropy loss. Because no public dataset exists for the task, the authors collect HDR images from video sources [45, 46, 47] for training and test on the LiU and HdM datasets. Evaluation uses PSNR and SSIM, with pixels near the image boundary excluded. Note that the paper states no real modulo-camera footage was used; data is prepared synthetically from HDR frames by quantizing to B bits and deriving A-bit modulo frames, folding numbers, and binary masks.

Why This Matters

Impact on research. Modulo sensing was introduced over a decade ago, but the paper states progress on modulo image recovery has been slow, especially in the use of modern deep learning. Existing methods (the MRF approach and UnModNet) work only on single images and lack temporal modeling. This work opens the video dimension, which the authors say had no prior work at all, and it contends that HDR techniques cannot simply be repurposed for it.

Real-world applications (as motivated in the paper):

• Automotive vision, where scenes span very wide light intensity ranges.
• Advanced displays.
• Photography, particularly for captured scenes with extreme brightness variation.
• Any video capture scenario where conventional sensors saturate — the paper frames modulo sensing as a route to theoretically unlimited dynamic range.

Industry relevance. The paper claims modulo sensors provide theoretically infinite dynamic range because the counter resets rather than saturating, which is attractive where well capacity and ADC precision limit conventional sensors. However, the framework depends on pre-training and has not yet been demonstrated on real modulo-camera hardware, so deployment is prospective rather than shown.

Future Directions

• Robustness to extreme folding. The authors explicitly list improving performance in highly folded regions as future work, since the current approach "may struggle with extreme folding cases."
• Real-world modulo cameras. Extending the framework from synthetically generated modulo data to actual modulo camera hardware is named as future work.
• Reducing dependence on pre-training. The paper states the approach "relies on pre-training" as a limitation, implying a need for less pre-training-heavy variants.
• Harder modulo patterns. Since all compared algorithms — including SSViT — struggle on disordered or abstract modulo patterns, closing that gap is an open problem.
• Additional analyis availability. The paper points to a supplementary site (https://sites.google.com/view/supplementaryijcnnid5522) for extended analyses and video comparisons, indicating further results exist beyond the main text.

Target Audience

Researchers and graduate students in computational photography, HDR imaging, and sensor design; computer vision practitioners working on video restoration and Transformer architectures; and engineers evaluating modulo sensors for automotive, display, or photography pipelines. Readers without a background in modulo arithmetic, HDR tone mapping, or Vision Transformers will find the methodology section demanding, though the problem framing and experimental tables are accessible.

Authors’ abstract

Conventional image sensors have limited dynamic range, causing saturation in high-dynamic-range (HDR) scenes. Modulo cameras address this by folding incident irradiance into a bounded range, yet require specialized unwrapping algorithms to reconstruct the underlying signal. Unlike HDR recovery, which extends dynamic range from conventional sampling, modulo recovery restores actual values from folded samples. Despite being introduced over a decade ago, progress in modulo image recovery has been slow, especially in the use of modern deep learning techniques. In this work, we demonstrate that standard HDR methods are unsuitable for modulo recovery. Transformers, however, can capture global dependencies and spatial-temporal relationships crucial for resolving folded video frames. Still, adapting existing Transformer architectures for modulo recovery demands novel techniques. To this end, we present Selective Spatiotemporal Vision Transformer (SSViT), the first deep learning framework for modulo video reconstruction. SSViT employs a token selection strategy to improve efficiency and concentrate on the most critical regions. Experiments confirm that SSViT produces high-quality reconstructions from 8-bit folded videos and achieves state-of-the-art performance in modulo video recovery.