HATIR: Heat-Aware Diffusion for Turbulent Infrared Video Super-Resolution

AI summary

Overview

Research area: Computer vision — video super-resolution (VSR) and atmospheric turbulence mitigation for infrared (thermal) video, using diffusion-based generative models.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, optical flow estimation, frequency-domain (phasor) analysis, deformable/warping operations, and video restoration evaluation metrics.

Scope: The paper introduces HATIR, a heat-aware latent diffusion framework that jointly removes atmospheric turbulence distortions and performs super-resolution on infrared video, together with FLIR-IVSR, the first dataset built for this joint task.

What This Paper Is About

Infrared video captured outdoors is degraded by atmospheric turbulence, which causes geometric distortions, thermal blur, and grayscale drift, on top of ordinary low-resolution compression and sampling loss. Existing video super-resolution methods were designed for visible-light imagery and ignore both the infrared modality gap and turbulence-induced distortions, while stacking a turbulence mitigation method in front of a VSR method causes errors to propagate and accumulate because the two degradations are modeled separately. HATIR instead injects heat-aware deformation priors directly into the diffusion sampling path so that alignment and restoration are solved as one unified inverse process.

Key Contributions

1. HATIR framework: A Heat-Aware Diffusion model for Turbulent InfraRed Video Super-Resolution that injects physics-driven heat-aware deformation priors into the reverse diffusion process, jointly modeling turbulence degradation and structural detail loss rather than treating them as separate stages.
2. Phasor-Guided Flow Estimator (PhasorFlow): A heat-aware flow estimator built on the physical observation that thermally active regions show consistent phasor (temporal frequency) responses, providing turbulence-aware motion guidance for reverse diffusion through a Phasor Mask, Frequency-Weighted Attention, and Heat-aware Guidance.
3. Turbulence-Aware Decoder (TAD): A decoder that suppresses unstable temporal cues and strengthens edge-aware feature aggregation using Turbulence Mask Gating (TMG) and IR Structure-Aware Attention (IR-SAA), to preserve structural fidelity under nonuniform distortions.
4. FLIR-IVSR dataset: Described as the first dataset for turbulent infrared VSR, containing paired LR-HR sequences captured with a FLIR T1050sc camera at 1024 × 768 resolution, spanning 640 diverse scenes with varying camera and object motion conditions.

Main Findings

• Best performance on FLIR-IVSR: On the FLIR-IVSR test set (camera-static/static scenes), HATIR reaches PSNR 33.3719, SSIM 0.8683, LPIPS 0.1227, DISTS 0.1183, and VMAF 46.6922, outperforming all compared methods across the reported camera-motion settings.
• Advantage on the M³FD turbulence benchmark: HATIR scores PSNR 29.7819 on the Set5 subset, 30.6093 on Set10, and 30.3834 on Set20, with the paper reporting clear advantages on the larger test sets (sizes 10 and 20).
• Two-stage pipelines accumulate errors: MambaTM, DATUM, and Turb-Seg, each combined with BasicVSR, suffer from error accumulation during turbulence removal that the subsequent super-resolution step amplifies.
• Visible-light VSR methods struggle: MIA-VSR, FMA-Net, EGOVSR, and IART fail to address noise, blurring, and spatial distortion caused by turbulence; the diffusion-based MGLDVSR recovers some blurred and noisy content but shows texture loss and restoration errors due to missing turbulence mitigation and infrared-specific guidance.
• PhasorFlow beats SpyNet: Replacing PhasorFlow with the pretrained SpyNet flow network drops performance from PSNR 33.6507 to 28.9387, with the paper reporting improvements of approximately 4.7 dB in PSNR and 12 points in VMAF, plus better preservation of object boundaries and thermally active regions such as human silhouettes.
• TAD components are complementary: With neither TMG nor IR-SAA, PSNR is 26.3283, SSIM 0.6775, LPIPS 0.2862, DISTS 0.1987, VMAF 32.0941. IR-SAA alone gives 27.3985 PSNR, TMG alone 28.4125 PSNR, and both together 32.2391 PSNR, 0.8229 SSIM, 0.1358 LPIPS, 0.1431 DISTS, 43.4152 VMAF. Removing IR-SAA hurts perceptual quality most; removing TMG increases LPIPS and DISTS.
• Joint masking beats single masks in guidance: With no mask, PSNR is 26.3073; occlusion mask only, 31.6965; Phasor Mask only, 28.9239; both combined into the joint heat-aware modulation mask, 32.1595 (SSIM 0.8087, LPIPS 0.1573, DISTS 0.1478, VMAF 42.6042).

Methodology in Plain English

HATIR treats turbulent infrared video restoration as one generative reverse-diffusion process instead of a chain of separate fix-up steps.

First, the low-resolution video is compressed into a latent representation by a VAE encoder. A PhasorFlow module then estimates motion between frames. Rather than relying on ordinary optical flow — which breaks down on low-contrast thermal imagery — it analyzes how each pixel's brightness oscillates over time in the frequency domain. Pixels whose brightness oscillates consistently (heat-emitting, thermally stable regions) get high weight in a soft "Phasor Mask," while pixels dominated by erratic turbulence get suppressed. This mask modulates both the warping used for alignment and the attention weights inside the flow network, and the flow is refined iteratively, locally in parallel and globally in a recurrent manner.

During diffusion sampling, the model adds a guidance term computed from the disagreement between forward and backward warped latents, weighted by a joint mask that combines the thermal Phasor Mask with an occlusion mask. This nudges each denoising step toward temporally coherent, thermally stable results.

Finally, the Turbulence-Aware Decoder rebuilds the high-resolution frames. It builds a disturbance heatmap from bidirectional warping errors to gate out turbulence-corrupted regions, then applies structure-aware attention based on gradient magnitude so that consistency is enforced mainly on edges and contours rather than flat, low-saliency areas. The decoder is fine-tuned on top of a pretrained VAE decoder using three losses: a thermal reconstruction loss, a thermal edge loss (using a Laplacian operator), and a frame difference loss for temporal consistency.

Training used an NVIDIA A800 GPU with the Adam optimizer (β1 = 0.9, β2 = 0.999); the U-Net backbone was initialized from Stable Diffusion v2.1 weights, and a lightweight time-aware encoder feeds LR temporal features in as conditioning.

Why This Matters

Impact on research: The paper argues that turbulence mitigation and super-resolution should not be decoupled, showing that two-stage pipelines propagate and accumulate errors. It contributes both a unified diffusion formulation with physically grounded thermal priors and a new benchmark dataset (FLIR-IVSR) for a task that previously had no dedicated data.

Real-world applications (as listed by the authors):

• Autonomous driving, where infrared sensing supports perception in poor visibility.
• Surveillance in low-visibility conditions.
• Thermal monitoring under challenging atmospheric conditions.
• General vision tasks in challenging environments such as object tracking.

Industry relevance: Infrared imaging is used in security, automotive night-vision, industrial inspection, and monitoring, all of which are affected by long-range atmospheric turbulence. A single model that removes turbulence and raises resolution reduces pipeline complexity compared with chaining separate mitigation and upscaling systems, and the released dataset gives industry and academic groups a shared benchmark. The project page is listed at https://github.com/JZ0606/HATIR.

Future Directions

• Extending FLIR-IVSR: The dataset is split into camera-moving (135 sequences) and camera-static (510 sequences, comprising 495 dynamic-scene and 15 static-scene sequences), and the authors state that building it is detailed in supplementary materials and that it encourages future infrared VSR research; broader scene coverage and motion conditions remain open.
• Bridging synthetic and real turbulence: Evaluation combines real FLIR-IVSR sequences with a synthetic turbulence benchmark built from static scenes of the public M³FD dataset; how well models transfer between synthetic and real turbulence is not resolved here.
• Efficiency of the diffusion pipeline: The paper reports no runtime, memory, or inference-cost comparisons, so whether this unified diffusion approach is practical for real-time or embedded infrared systems is not reported.
• Downstream task validation: The authors motivate the work with autonomous driving, surveillance, and tracking, but no downstream task evaluation (detection, tracking accuracy) is reported, leaving the practical benefit of the restoration unmeasured.

Target Audience

Researchers and graduate students working on video super-resolution, diffusion-based generative models, infrared/thermal imaging, and atmospheric turbulence mitigation; also engineers building infrared perception or surveillance systems who need to understand the tradeoffs between cascaded turbulence-mitigation-plus-super-resolution pipelines and unified approaches, and dataset builders interested in the FLIR-IVSR benchmark.

Authors’ abstract

Infrared video has been of great interest in visual tasks under challenging environments, but often suffers from severe atmospheric turbulence and compression degradation. Existing video super-resolution (VSR) methods either neglect the inherent modality gap between infrared and visible images or fail to restore turbulence-induced distortions. Directly cascading turbulence mitigation (TM) algorithms with VSR methods leads to error propagation and accumulation due to the decoupled modeling of degradation between turbulence and resolution. We introduce HATIR, a Heat-Aware Diffusion for Turbulent InfraRed Video Super-Resolution, which injects heat-aware deformation priors into the diffusion sampling path to jointly model the inverse process of turbulent degradation and structural detail loss. Specifically, HATIR constructs a Phasor-Guided Flow Estimator, rooted in the physical principle that thermally active regions exhibit consistent phasor responses over time, enabling reliable turbulence-aware flow to guide the reverse diffusion process. To ensure the fidelity of structural recovery under nonuniform distortions, a Turbulence-Aware Decoder is proposed to selectively suppress unstable temporal cues and enhance edge-aware feature aggregation via turbulence gating and structure-aware attention. We built FLIR-IVSR, the first dataset for turbulent infrared VSR, comprising paired LR-HR sequences from a FLIR T1050sc camera (1024 X 768) spanning 640 diverse scenes with varying camera and object motion conditions. This encourages future research in infrared VSR. Project page: https://github.com/JZ0606/HATIR