Enhancing Infrared Vision: Progressive Prompt Fusion Network and Benchmark

AI summary

Overview

Research area: Computer vision, specifically thermal infrared (TIR) image enhancement and all-in-one image restoration.

Technical level: Advanced. The paper defines a formal degradation model, prompt-conditioned feature modulation, and a multi-stage training scheme; it assumes familiarity with restoration backbones such as Restormer, NAFNet, UFormer, XRestormer, and FocalNet.

Scope (one sentence): The paper proposes a Progressive Prompt Fusion Network (PPFN) with a Selective Progressive Training (SPT) scheme for thermal infrared enhancement under single and composite degradations, and introduces a new multi-scenario benchmark called HM-TIR containing 1,503 TIR images at 640 × 512.

What This Paper Is About

Thermal infrared images are degraded by several problems at once — low contrast, blurring, and sensor noise — yet most existing enhancement methods are built to fix only one of these at a time. Methods designed for visible-light images also transfer poorly because TIR imaging follows a different physical process. The paper's goal is a single model that handles thermal-specific degradations, individually or combined, plus a large, diverse benchmark to measure progress on that task.

Key Contributions

1. A Progressive Prompt Fusion Network (PPFN) for TIR enhancement that targets hybrid degradations. The authors state this is the first study to address TIR enhancement under such multifaceted degradation conditions.
2. A prompt fusion block that injects prior knowledge (degradation type and scenario type) into the learning process via dual prompts — degradation-specific and type-specific — and is described as a plug-and-play module that can be inserted into existing network architectures. It was tested with five models: FocalNet, UFormer, NAFNet, XRestormer, and Restormer.
3. A Selective Progressive Training (SPT) mechanism that handles composite and single degradations differently: for composite cases, outputs from one stage feed the next so degradations are removed in reverse order; for single degradations, standard training is used.
4. The HM-TIR benchmark, described as a high-quality multi-scenario TIR dataset of 1,503 images at 640 × 512 within the 8 to 14 micrometer wavelength range, covering scenes such as road, square, city, forest, campus, coastline, residential zone, and others, with five corruption types: low contrast, blur, stripe noise, optical noise, and Gaussian noise.

Main Findings

• Composite degradation performance: The proposed approach reports a notable 8.76% improvement on the Normal Set in PSNR. Both quantitative and qualitative results show PPFN outperforming compared methods on multi-degradation TIR enhancement.
• Comparison against TIR-specific methods: WFAF, LRSID, and TIRIE (listed as TSIRIE) show lower PSNR and SSIM because they are tailored to single degradations; qualitatively they produce substantial artifacts and background noise.
• Comparison against visible all-in-one methods: DA-CLIP and DiffUIR perform better than the TIR-specific single-degradation methods but still show noticeable blurring and distortion because of imaging-model differences between visible and infrared spectra.
• Real-world Iray dataset results (no-reference IQA): The proposed method reports the best NIMA (3.8327), best MUSIQ (30.9072), and best NIQE (8.4693, lower is better). For context, the degraded input scores 3.5326 / 25.2459 / 10.1277, and the strongest competing values shown are DA-CLIP's NIMA of 3.7004 and MUSIQ of 29.0350, LRSID's NIQE of 8.6838, and DiffUIR's MUSIQ of 27.7855 and NIQE of 9.1896. Other compared methods are WFAF, TIRIE (TSIRIE), IE-CGAN, and the Restormer baseline (3.5812 / 27.7829 / 8.7776).
• Single-degradation results: The method is reported to reduce noise, recover fine detail, and enhance contrast better than denoising methods (AP-BSN, CycleISP, IDR, SDAP), deblurring methods (DeBlurGANv2, MIMO-UNet, FFTformer, Stripformer), and contrast-enhancement methods (dehazing: MSBDN, FFA-Net; low-light: LLFormer, SCI).
• Backbone-agnostic gains: Adding PPFN and SPT improves every tested backbone. In the reported average rows, Restormer goes from 23.28/0.796 to 25.32/0.818, XRestormer from 23.54/0.801 to 24.75/0.811, NAFNet from 22.29/0.776 to 23.74/0.792, and FocalNet from 21.22/0.778 to 22.63/0.790 on the Normal Set; UFormer is 21.95/0.775 without and 21.62/0.768 with. The paper reports a second row of averages for its other test set (for example Restormer 22.87/0.757 to 23.27/0.764).
• Ablation on prompts: Using degradation-specific prompts gives a 0.29 dB gain over the baseline; using both type- and degradation-specific prompts gives 0.40 dB. Removing the non-linear activation or replacing concatenation with multiplication causes a PSNR decline, with multiplication giving only minimal SSIM improvement.
• Ablation on SPT: Applying iterative training directly to the baseline causes a PSNR drop of 0.23 dB (22.8678 to 22.6357). With the full PPFN, results improve as iterations increase: 1 iteration gives 14.5455/0.6125, 2 iterations 14.6080/0.6261, and 3 iterations 23.2712/0.7643.
• Prompt and order sensitivity: Using a single-degradation prompt on composite degradation fails to remove the degradation, leaving artifacts. Incorrect degradation-removal order yields lower PSNR, supporting the fixed-order assumption.

Methodology in Plain English

The authors start from a physical description of thermal imaging degradation and write it as an ordered composition: low contrast is applied first, then blur, then fixed-pattern noise (stripe and optical noise), with additive random noise on top. Inference reverses this order — denoising, then deblurring, then decontrast.

To make one model handle many degradations, they give it two kinds of learned "prompts": one set that tells the model which degradation to deal with (noise, blur, or contrast) and another that tells it whether it is facing a single or a composite case. Two lightweight encoders turn these prompts into features, which are concatenated, passed through a linear layer and a non-linear activation, and then converted into two channel-wise modulation parameters. Those parameters rescale and shift each layer's features of the restoration network, so the network's behaviour is steered per degradation type and per scenario.

Training follows a selective progressive scheme. For composite degradations, the fully degraded image is used as input and the network is asked to undo one degradation per iteration, with the ground truth for each iteration being the output of the previous degradation step; gradients are accumulated across iterations and the parameters updated once at the end. Stop-gradient is applied to the intermediate output that becomes the next iteration's input to reduce training cost. For single degradations, ordinary training with the clean image as ground truth is used.

Experiments use Restormer as the main baseline, implemented in PyTorch on four 4090D GPUs. Degradation is synthesized with a Gated Degradation pipeline with all gate probabilities set to 0.8. Training uses an L1 loss and Adam with beta1 = 0.9 and beta2 = 0.999, batch size 4, random cropping and flipping at 256 × 256 patches, an initial learning rate of 8 × 10⁻⁵ decaying to 10⁻⁶ with a cosine annealing schedule, and 300 epochs. The 1,503-image HM-TIR set is split 80% training / 20% validation, with a Normal Set and a Hard Set for multi-degradation testing and three single-degradation test subsets built with the Hard Set settings.

Why This Matters

Impact on research: The work reframes TIR enhancement as a composite-degradation problem rather than a collection of single-degradation tasks, and it supplies a benchmark (HM-TIR) that the authors argue is higher quality and more varied than prior TIR sets. Table 1 contrasts HM-TIR (2025, image format, 1,503 images, 640 × 512, multiple camera angles, scenes ①–⑧, corruptions I–V) against EN (2019, 16 images, 256 × 256, scene ④⑤⑦, corruption I), Iray (2021, 2,000 images, 256 × 192, scene ①⑧, corruption III), SBTI (2022, 4 videos, 640 × 480, scenes ①③, corruption II), UIRD (2023, 17 videos, 640 × 512, scenes ①③, corruption II), and TIVID (2024, 518 videos, 320 × 256, scenes ①③④⑦, corruptions III, IV, V). The prompt fusion block is also presented as plug-and-play, which lowers the barrier to reusing it in other restoration architectures.

Real-world applications (as identified in the paper):

• Object detection in thermal imagery.
• Semantic segmentation in thermal imagery.
• Autonomous driving, where thermal sensing works without external light.
• Imaging in complete darkness, low-light conditions, and through smoke, haze, and minor obstructions.

Industry relevance: Thermal cameras are constrained by expensive specialized materials such as Mercury Cadmium Telluride (MCT) and Indium Antimonide (InSb), and are sensitive to temperature fluctuations and atmospheric conditions. A software enhancement layer that cleans up low-quality thermal captures could reduce the cost pressure of obtaining high-quality TIR imagery and extend the usefulness of existing sensor hardware.

Future Directions

• The paper

Authors’ abstract

We engage in the relatively underexplored task named thermal infrared image enhancement. Existing infrared image enhancement methods primarily focus on tackling individual degradations, such as noise, contrast, and blurring, making it difficult to handle coupled degradations. Meanwhile, all-in-one enhancement methods, commonly applied to RGB sensors, often demonstrate limited effectiveness due to the significant differences in imaging models. In sight of this, we first revisit the imaging mechanism and introduce a Progressive Prompt Fusion Network (PPFN). Specifically, the PPFN initially establishes prompt pairs based on the thermal imaging process. For each type of degradation, we fuse the corresponding prompt pairs to modulate the model's features, providing adaptive guidance that enables the model to better address specific degradations under single or multiple conditions. In addition, a Selective Progressive Training (SPT) mechanism is introduced to gradually refine the model's handling of composite cases to align the enhancement process, which not only allows the model to remove camera noise and retain key structural details, but also enhancing the overall contrast of the thermal image. Furthermore, we introduce the most high-quality, multi-scenarios infrared benchmark covering a wide range of scenarios. Extensive experiments substantiate that our approach not only delivers promising visual results under specific degradation but also significantly improves performance on complex degradation scenes, achieving a notable 8.76\% improvement. Code is available at https://github.com/Zihang-Chen/HM-TIR.