Revisiting Generative Infrared and Visible Image Fusion Based on Human Cognitive Laws

AI summary

arXiv: 2510.26268v1 [cs.CV], 30 Oct 2025 Authors: Lin Guo, Xiaoqing Luo, Wei Xie, Zhancheng Zhang, Hui Li, Rui Wang, Zhenhua Feng, Xiaoning Song Code: https://github.com/lxq-jnu/HCLFuse

Overview

Research area: Computer vision, specifically generative multimodal image fusion (infrared plus visible imagery), with connections to information theory, optimal transport, and diffusion-based generative modeling.

Technical level: Advanced. The paper is built on information bottleneck theory, second-order Wasserstein distance, optimal transport plans, variational inference, and denoising diffusion sampling. The main text is readable with background knowledge, but the derivations in the appendices require graduate-level probability and optimization.

Scope: One paper that proposes HCLFuse, a generative infrared/visible fusion framework that combines a variational bottleneck encoder with a physics-guided diffusion process inspired by human cognitive laws, and evaluates it on multiple fusion benchmarks plus a downstream segmentation task.

What This Paper Is About

Infrared and visible image fusion combines thermal information from an infrared sensor with textural detail from a visible sensor into a single image. Existing generative methods are criticized here on three grounds: their generative capability is limited (they mostly extract or optimize features rather than fully exploit generation), the way they select information from each modality is not interpretable, and they depend heavily on data statistics, which makes them fragile under distribution shifts or noise. The goal of this work is to make fusion both more interpretable and more structurally consistent by borrowing two principles from human cognition: selective attention (keep task-relevant information, discard redundancy) and adherence to physical laws (combine perceptual input with domain knowledge).

Key Contributions

1. A cognition-inspired generative fusion framework (HCLFuse). It integrates a data-driven generative model with theorem-constrained probabilistic reasoning to improve modality interpretability and structural consistency, departing from fusion models that treat the task purely as a data-distribution problem.

2. A multi-scale variational bottleneck encoder (VBE). Built on an unsupervised information-mapping quantization theory, it uses posterior probability modeling and information decomposition to extract compact, discriminative low-level modal representations that support high-fidelity structural detail generation. The paper derives a lower bound on task-relevant mutual information via an optimal transport mapping (Theorem 1) and an upper bound on redundant mutual information in the perturbation term (Theorem 2).

3. A time-varying physical guidance (TPG) mechanism. The probabilistic generative ability of a diffusion model is combined with physical laws so that the generation process is adaptively regulated at different stages, improving the model's ability to perceive the intrinsic structure of data and reducing dependence on data quality. Three physical constraints are used: heat conduction, structure preservation, and physical consistency.

4. Extensive evaluation. Qualitative and quantitative comparisons against 17 competing methods, plus ablation studies and a downstream semantic segmentation evaluation.

Main Findings

• Curvature-based structural diagnosis: Gaussian curvature is used as a geometric indicator of structural consistency. The paper reports that existing diffusion-based methods show fragmented curvature patterns in critical regions, indicating incoherent structure retention and biased information integration, whereas HCLFuse shows more complete structural features in curvature visualization and in certain regions surpasses the curvature quality of the original source images.

• MSRS quantitative results (Table 1): HCLFuse obtains the best values on most reported metrics with SD 49.546, AG 6.4355, CC 0.6186, SCD 1.6575, EN 6.8704, SF 17.899, Nabf 0.0017, DF 7.6427, QSF 0.5374, VIF 0.8540, PIQE 25.193, and BRI 26.215. The paper describes HCLFuse as achieving superior performance on most metrics, excelling in texture clarity and structural fidelity. In Table 1, the highest SD among all listed methods is Text-DiFuse at 54.243, and the highest CC is DDFM at 0.6585, so the "best on most metrics" claim is not a claim of best on every column.

• Reported relative gains on MSRS: AG is reported as outperforming the second-best method by 69.87 percent, SF shows a 39.41 percent relative gain, and DF improves on the next best result by 65.56 percent. The paper additionally states that HCLFuse achieves a substantially higher QSF score than all competing methods and the highest EN, which it attributes to preserved directionally distributed frequency information and information richness with suppressed unnatural responses.

• Qualitative daytime results (Figure 3, MSRS scene 00621D): FusionGAN, SOSMaskFuse, CrossFuse, and CCF are described as over-enhancing thermal response with unnatural brightness and artifacts; MMAE is described as failing to preserve a sign within the green box, a critical structural loss. HCLFuse is reported to preserve complementary features from both modalities with detail restoration, including bicycle wheels, background leaves, and pavement textures.

• Qualitative nighttime results (Figure 4, MSRS scene 00774N): LRRNet, CrossFuse, and GIFNet are described as failing to maintain thermal prominence of pedestrians in the red box. TarDAL, Text-DiFuse, and HCLFuse reportedly preserve structural information in the green-box region, with only HCLFuse maintaining higher perceptual resolution with clearer details.

• Ablation results (Table 2): The full model reaches SD 49.55, AG 6.436, CC 0.619, SCD 1.658, EN 6.870, SF 17.899, Nabf 0.0017, DF 7.643, QSF 0.537, VIF 0.854, PIQE 25.19, and BRI 26.22. Removing TPG ("W/O TPG") drops most metrics (SD 36.90, AG 5.521, SF 15.158, DF 6.513) even though CC (0.646) and Nabf (0.0015) are slightly better than the full model. Removing the VBE ("W/O VBE", replaced with a standard multi-scale encoder) ranks second overall but produces visible artifacts such as coarse building textures and unnatural sky transitions. Removing optimal transport ("W/O OT") causes sharp declines across all metrics (SD 28.66, AG 3.578, SF 11.090). Removing DDIM ("W/O DDIM") degrades both quantitative scores and visual quality (SD 28.36, AG 3.626, SF 11.218).

• Generalization: The paper states that additional comparisons on the TNO and FMB datasets show HCLFuse consistently outperforming existing fusion methods with strong generalization and robustness under varying conditions. The corresponding quantitative and qualitative results are placed in Appendix C and are not given in the main text.

• Downstream segmentation: Comparative experiments on MSRS using the Mask2Former framework reportedly show HCLFuse achieving superior segmentation performance, attributed to retained fine structural and semantic cues. The specific segmentation numbers are in Appendix C.4 and are not reported in the main text.

• Acknowledged limitations: The method relies on well-aligned infrared and visible image pairs, and the diffusion process introduces computational overhead, which may limit applicability in real-time or resource-constrained scenarios.

Methodology in Plain English

The method has three main stages.

1. Align the two modalities before fusing them. Infrared and visible images of the same scene can differ structurally. The authors treat this as an optimal transport problem: they compute an optimal transport plan that maps the flattened infrared image onto the visible modality, and only the infrared image is transformed while the visible image stays unchanged for stability and efficiency. Theorem 1 relates this to a lower bound on task-relevant mutual information, showing that the information gain is bounded below by the reduction in Wasserstein distance after applying the transport map. This gives the unsupervised model an optimizable objective despite having no labels.

2. Compress into a compact latent code with a variational bottleneck encoder. The aligned infrared image and the original visible image are concatenated and passed through an encoder. Learnable per-scale masks, produced by applying a sigmoid to learnable weights, select which features matter, embodying the "selective attention" idea. The encoder's posterior is modeled as a Gaussian, and the latent variable is decomposed into a deterministic mean term (task-relevant structure) plus a zero-mean Gaussian perturbation (uncertainty from compression). The training objective combines reconstruction of both inputs with a Kullback-Leibler divergence term that keeps the posterior near the prior, controlled by a trade-off coefficient. Theorem 2 gives an upper bound on redundant mutual information between the perturbation and the mean, which is meant to keep the diffusion stage focused on task-relevant structure.

3. Generate with a diffusion process corrected by physical laws. At each denoising timestep the model first produces a probabilistic estimate of the clean latent, then applies a physics-based correction, then performs reverse sampling with a re-estimated noise term so that both semantic structure and noise stay consistent. Three corrections are applied in sequence: a heat conduction constraint using the Laplacian operator and a time-dependent diffusion coefficient, a structure preservation constraint that pulls gradients toward the maximum gradient map of the source pair inside a structural mask derived from high-frequency visible responses, and a physical consistency constraint that blends masked infrared and visible contributions with weights for each modality. Each constraint's strength follows a decaying schedule (initial weight times an exponential decay with a decay factor over the normalized timestep), so constraints are strong early when noise and uncertainty are high and weaker later. The authors describe this as mirroring "coarse perception followed by fine reasoning."

Evaluation uses four public datasets as stated in the paper, with MSRS, TNO and FMB named, 17 comparison methods, seven no-reference and five reference-based metrics, and an NVIDIA RTX 3090 GPU. All components are implemented on top of a DDIM deterministic sampling scheme.

Why This Matters

Impact on research. The paper reframes generative fusion as a problem of interpretable information selection rather than only distribution matching. It supplies theoretical grounding for unsupervised fusion through an optimal-transport lower bound and a redundancy upper bound, and it demonstrates that injecting non-learnable physical priors into a diffusion trajectory can improve structural consistency. The Gaussian curvature diagnostic shown in Figure 1 is also a distinctive way of visualizing whether a fusion method really preserves structure.

Real-world applications (as framed by the paper):

• Surveillance systems using multi-modal sensing.
• Autonomous driving, where thermal cues and visible texture complement each other.
• Target tracking under varying conditions.
• Nighttime and adverse-weather scenarios, since the evaluation spans urban driving, nighttime military scenes, and adverse weather, and infrared sensing is effective in low light where visible sensors degrade.

Industry relevance. Because the paper reports gains on downstream semantic segmentation with Mask2Former, the work speaks directly to perception pipelines where fusion is a pre-processing step feeding segmentation or detection. The explicit acknowledgment of diffusion computational overhead and dependence on aligned image pairs gives practitioners a clear sense of where deployment would be difficult, particularly for real-time or resource-constrained devices.

Future Directions

• Lowering the computational cost. The authors identify diffusion overhead as a limitation for real-time and resource-constrained use; distillation, fewer sampling steps, or lighter encoders are natural follow-ups.
• Handling unaligned or misregistered modality pairs. The current formulation depends on well-aligned infrared and visible pairs, and the optimal transport step assumes a pointwise structure between flattened images. Extending to unaligned capture would broaden applicability.
• Broadening the physics-informed constraints. Only heat conduction, structure preservation, and physical consistency are used here; other physical laws, or learned masks replacing the hand-designed heat and structural masks, are open questions, especially since the masks are described as non-learnable spatial priors.
• Stress-testing the robustness claim. The paper argues that physics guidance reduces dependence on data quality, but the reported evidence for data distribution shifts and noise interference is qualitative. Systematic noise and shift benchmarks would test this directly, as would full reporting of the TNO, FMB, and segmentation numbers that currently sit in the appendix.

Target Audience

Researchers and advanced graduate students working on multimodal image fusion, diffusion and other generative models for low-level vision, and information-theoretic representation learning. It is also relevant to applied engineers building perception systems for surveillance, autonomous driving, or nighttime and adverse-weather operation who need to understand both the reported accuracy gains and the practical constraints of a diffusion-based fusion pipeline.

Authors’ abstract

Existing infrared and visible image fusion methods often face the dilemma of balancing modal information. Generative fusion methods reconstruct fused images by learning from data distributions, but their generative capabilities remain limited. Moreover, the lack of interpretability in modal information selection further affects the reliability and consistency of fusion results in complex scenarios. This manuscript revisits the essence of generative image fusion under the inspiration of human cognitive laws and proposes a novel infrared and visible image fusion method, termed HCLFuse. First, HCLFuse investigates the quantification theory of information mapping in unsupervised fusion networks, which leads to the design of a multi-scale mask-regulated variational bottleneck encoder. This encoder applies posterior probability modeling and information decomposition to extract accurate and concise low-level modal information, thereby supporting the generation of high-fidelity structural details. Furthermore, the probabilistic generative capability of the diffusion model is integrated with physical laws, forming a time-varying physical guidance mechanism that adaptively regulates the generation process at different stages, thereby enhancing the ability of the model to perceive the intrinsic structure of data and reducing dependence on data quality. Experimental results show that the proposed method achieves state-of-the-art fusion performance in qualitative and quantitative evaluations across multiple datasets and significantly improves semantic segmentation metrics. This fully demonstrates the advantages of this generative image fusion method, drawing inspiration from human cognition, in enhancing structural consistency and detail quality.