MagicFuse: Single Image Fusion for Visual and Semantic Reinforcement

AI summary

Overview

Research area: Computer vision, specifically infrared–visible image fusion (IVIF), diffusion-based generative modeling, and semantic segmentation for machine perception.

Technical level: Advanced. The work assumes familiarity with latent diffusion models, DDIM sampling, noise-prediction objectives, and fusion quality metrics (EN, SSIM, MI, Qabf, PSNR, mIoU).

Scope: The paper proposes and evaluates MagicFuse, a framework that generates a joint visible–infrared scene representation from a single degraded visible image, replacing data-level multi-modal fusion with what the authors call knowledge-level fusion.

What This Paper Is About

Infrared–visible image fusion normally requires a thermal infrared camera and a visible camera to be recording the same scene at the same time, which is often impossible because thermal sensors are expensive. The authors ask whether the benefits of fusion can still be obtained when only a visible sensor is present, and the visible image is degraded by low light, haze, or noise.

Their goal is a framework that infers both the information hidden inside the visible spectrum and the thermal radiation patterns that an infrared sensor would have captured, then merges these two sources of "knowledge" into one output image, called the Magic Image (MagImg), that serves both human viewing and downstream semantic tasks.

Key Contributions

1. Introduces Single Image Fusion (SIF). The paper states this is the first proposal of SIF, a setting in which a comprehensive cross-spectral scene representation is derived from a single low-quality visible image, with no infrared input at any stage.

2. Bridges data-level and knowledge-level fusion. Two diffusion branches — intra-spectral knowledge reinforcement (IKR) and cross-spectral knowledge generation (CKG) — are combined by a multi-domain knowledge fusion (MKF) branch that merges the probabilistic noise estimated by both branches at each timestep.

3. Couples visual and semantic constraints. An auxiliary segmentation head embedded in the MKF branch aligns fused features with semantic labels, and a radiation category map derived from the predicted segmentation modulates the fusion weighting, which the authors also identify as a fix for a training collapse problem in the noise fusion step.

4. Demonstrates competitive performance with less input. Experiments report that MagicFuse, using only a single degraded visible image, reaches visual and semantic performance comparable to or better than state-of-the-art fusion methods that consume paired infrared–visible inputs.

Main Findings

• Visual quality on MFNet test set (392 degraded visible images): MagicFuse records EN 7.29, SSIM 0.45, MI 4.13, Qabf 0.50, and PSNR 63.49. It records the highest EN (7.29 versus 7.26 for DAFusion) and the highest MI (4.13 versus 3.46 for EMMA) and the highest PSNR (63.49 versus 63.38 for Text-IF), while Text-IF records higher SSIM (0.52 versus 0.45) and higher Qabf (0.59 versus 0.50). All nine competitors use infrared–visible pairs.
• Semantic quality on MFNet: MagicFuse's fusion-network output (labeled Ours-F) reaches mIoU 62.19, second only to SegMiF at 62.28 and above the degraded visible baseline at 54.09. The standalone segmentation-head output (Ours-S) reaches mIoU 57.11.
• Generalization to FMB (280 test images): Visual metrics are EN 6.93, SSIM 0.40, MI 4.09, Qabf 0.57, PSNR 62.51; the paper reports second-best scores on most metrics on this unseen dataset. Semantic mIoU for Ours-F is 55.41 versus SegMiF at 57.58 and the degraded visible baseline at 53.39, so multi-modal methods are not surpassed here even though both the degraded and enhanced visible images are outperformed.
• Beyond fusion datasets (Cityscapes): 490 visible images are enhanced into MagImgs to retrain SegFormer and evaluation is performed on 154 MagImgs; segmentation mIoU rises from 65.80 to 66.91.
• Hyperparameter sensitivity: Sweeping τ over 0, 0.2, 0.4, 0.6, and 0.8 shows both MagImg quality and segmentation-head accuracy peak at τ = 0.4, which the authors interpret as evidence that cross-spectral knowledge must be regulated so it does not disrupt the original spectral information.
• Ablation results: Removing the single-step latent estimates from the fusion-weight computation (Model I), removing the segmentation head (Model II), aggregating branch outputs after diffusion rather than fusing noise per timestep (Model III), and using only IKR-enhanced visible images (Model IV) all reduce visual and semantic scores relative to the full model, whose metrics are EN 7.29, SSIM 0.45, MI 4.13, Qabf 0.50, PSNR 63.49, and mIoU 62.19.

Methodology in Plain English

The system treats the missing infrared information as something that can be inferred rather than measured. Two separate diffusion models are trained. The first, the IKR branch, learns image restoration: given a degraded visible image, it recovers the color, texture, and details that were obscured. The second, the CKG branch, learns visible-to-infrared translation: it learns from large paired datasets what thermal radiation patterns look like for objects such as pedestrians and cars, so it can predict a plausible infrared-style representation from a visible image alone.

Both branches start from the same random Gaussian noise and denoise step by step. At every step, each branch predicts a noise component that implicitly encodes its own kind of knowledge. The MKF branch then sits on top of those two streams and computes a per-step weighting that decides how much of each branch's predicted noise should be used. The weighting network looks at the two branches' one-step estimates of the clean image, the current noisy state of the fusion stream, the two noise predictions, and the timestep, and produces the blend. Training is done in two phases: the two diffusion models are optimized independently first, then frozen, after which the fusion network and segmentation head are trained using the signals from both streams.

Semantic information enters in two ways. A segmentation head reads the fusion network's attention features and predicts a segmentation map, trained with a cross-entropy loss. That map is converted into a radiation category map that adjusts the fusion weights so thermal objects are preserved. This modulation also prevents a collapse mode the authors describe: because both branches share the same input conditions, their noise predictions are correlated and the fusion weight would otherwise drift to 1, favoring the restoration branch entirely. Visual losses on luminance contrast, gradient texture, and chrominance keep the output faithful to the intended appearance.

Why This Matters

• Research impact: The paper reframes fusion as a knowledge-transfer problem rather than a sensor-pairing problem, answering two questions it poses: where new knowledge beyond the visible spectrum comes from (generative models trained on large-scale data) and how fusion can proceed without paired data (fusing estimated noise in diffusion streams instead of fusing images). This opens a direction in which fusion methods are no longer invalidated by the absence of a thermal sensor.

• Real-world applications:

  • Individual reconnaissance and surveillance in low-light, hazy, or noisy environments where only visible cameras are deployed.
  • Intelligent transportation and assisted driving, where pedestrians must be made salient without a thermal camera on the vehicle.
  • Low-cost machine perception for drones or robots, extending scene representation to platforms that cannot carry expensive infrared sensors.
  • General photo and video enhancement, as demonstrated by the Cityscapes experiment on ordinary natural images.

• Industry relevance: The framework requires no infrared hardware at inference, which lowers the sensor cost barrier for any deployment relying on fusion today. The code is publicly available at https://github.com/zhayanping/MagicFuse. Inference uses accelerated deterministic DDIM sampling with 25 steps, which is the kind of detail that matters when deciding whether such a model can run in a real pipeline.

Future Directions

• Closing the generalization gap. On the unseen FMB dataset, MagicFuse does not surpass multi-modal fusion methods on semantic mIoU (55.41 versus 57.58 for SegMiF), and on visual metrics several competitors score higher. Extending the knowledge-fusion training beyond the single MFNet training set used here is an obvious next step.

• Reducing model cost. The CKG branch alone has 517.67M parameters and 773.7 GFLOPs, compared with 39.65M for IKR and 2.74M for the fusion and segmentation modules; the paper does not report inference latency or memory for the full pipeline, so efficiency is an open question.

• Broadening modality coverage. The current design covers visible and infrared only. Whether the noise-fusion principle extends to other absent modalities is not addressed.

• Validating physical plausibility. Because the infrared content is generated rather than sensed, the paper does not report any study of whether the generated thermal patterns correspond to real physical measurements, which would matter for safety-critical use.

Target Audience

Researchers and graduate students working on image fusion, diffusion-based restoration and translation, and segmentation for autonomous or surveillance systems. Practitioners building perception pipelines for platforms where thermal sensors are unavailable or too costly will find the problem framing and the Cityscapes demonstration most directly useful. Readers without a background in latent diffusion models will need to consult the cited diffusion literature first, since the methodology sections assume it.

Authors’ abstract

This paper focuses on a highly practical scenario: how to continue benefiting from the advantages of multi-modal image fusion under harsh conditions when only visible imaging sensors are available. To achieve this goal, we propose a novel concept of single-image fusion, which extends conventional data-level fusion to the knowledge level. Specifically, we develop MagicFuse, a novel single image fusion framework capable of deriving a comprehensive cross-spectral scene representation from a single low-quality visible image. MagicFuse first introduces an intra-spectral knowledge reinforcement branch and a cross-spectral knowledge generation branch based on the diffusion models. They mine scene information obscured in the visible spectrum and learn thermal radiation distribution patterns transferred to the infrared spectrum, respectively. Building on them, we design a multi-domain knowledge fusion branch that integrates the probabilistic noise from the diffusion streams of these two branches, from which a cross-spectral scene representation can be obtained through successive sampling. Then, we impose both visual and semantic constraints to ensure that this scene representation can satisfy human observation while supporting downstream semantic decision-making. Extensive experiments show that our MagicFuse achieves visual and semantic representation performance comparable to or even better than state-of-the-art fusion methods with multi-modal inputs, despite relying solely on a single degraded visible image. The code is publicly available at https://github.com/zhayanping/MagicFuse.