PCDFusion: Proposal-Context-Detail Bayesian Rendering for Infrared-Visible Image Fusion

AI summary

Overview

• Research area: Computer vision, specifically infrared–visible image fusion (IVIF) and its downstream use in semantic segmentation and object detection.
• Technical level: Advanced — the paper builds on Bayesian odds-form posterior updates, cross-attention Transformer backbones, and multi-term unsupervised loss design.
• Scope (one sentence): The paper reframes infrared–visible fusion as a sequence of local Bayesian posterior rendering decisions (proposal, context, detail) and evaluates the resulting visible-anchored network on the MSRS, FMB and M3FD benchmarks plus downstream segmentation and detection.

What This Paper Is About

Visible cameras capture reflected light and provide natural texture, but they fail under poor or extreme exposure; thermal infrared cameras capture emitted radiance and stay robust to darkness, but lack natural texture and reliable fine structure. The paper argues that existing deep fusion methods mix the two modalities in a largely symmetric way, with no explicit local rule for deciding when visible appearance should dominate and when thermal evidence should contribute. PCDFusion's goal is to make that decision explicit by treating each local fusion operation as a posterior action weight computed from a physical prior plus a learned log-evidence correction, so thermal information is admitted only where visible observations are unreliable.

Key Contributions

1. A probabilistic reformulation of IVIF. Fusion is cast as local Bayesian posterior rendering: for each rendering operation a binary latent variable decides whether a stage-specific action is accepted, a physical cue forms the prior, and a learned log-evidence score (log-odds correction) shifts the prior to a normalized posterior action weight, which then mixes a base outcome with an action-conditioned outcome.

2. The PCDFusion network. A visible-anchored architecture with three ordered posterior rendering stages: a positive infrared proposal stage that keeps candidate thermal responses available, a visible context update stage that repairs exposure-degraded visible regions, and a detail stage that renders only band-limited infrared structures into the final luminance.

3. Stage-specific priors and evidence estimators. Three physically motivated priors — a normalized-infrared thermal-response prior, a symmetric U-shaped exposure-failure prior derived from visible luminance, and a structure-ratio prior comparing band-limited infrared structural support against visible structural support — each combined with a neural log-evidence estimator.

4. Label-free training with downstream validation. A three-term objective (proposal recall, context posterior calibration with a keep term, final gradient-magnitude fidelity) is used without semantic labels for fusion training, and the paper reports gains in image-level fusion metrics and in downstream segmentation (SegFormerB2) and detection (YOLOv10s) under challenging illumination.

Main Findings

• Consistent image-level gains on three benchmarks. Trained only on the MSRS training split and evaluated on all datasets without dataset-specific fine-tuning, PCDFusion reports MSRS AG 3.84 / SCD 1.67 / VIF 1.12 / Qabf 0.72 / Qcb 0.59; FMB AG 4.33 / SCD 1.57 / VIF 1.08 / Qabf 0.71 / Qcb 0.49; M3FD AG 5.04 / SCD 1.58 / VIF 0.95 / Qabf 0.67 / Qcb 0.50. The paper states PCDFusion leads in VIF, Qabf and Qcb across all three datasets, while some baselines obtain higher AG or SCD on individual datasets (for example Deno-IF reaches SCD 1.79 on MSRS and EMMA reaches AG 5.33 on M3FD).

• Best aggregate downstream scores. PCDFusion reports 60.05% mIoU on FMB segmentation, 1.07 percentage points above Diff-IF (58.98%), and 55.98% mAP 50:95 on M3FD detection, 0.19 percentage points above DCEvo (55.79%). The paper notes that no method ranks first on every individual class.

• All three rendering stages contribute. Removing the proposal stage produces the most evident degradation (AG 3.810, SCD 1.594, mIoU 58.44, mAP 55.18 versus the full model's 3.839, 1.671, 60.05 and 55.98); removing context weakens reconstruction despite raising SCD (mIoU 58.86, mAP 54.88); removing detail lowers the final balance (mIoU 58.89, mAP 56.13).

• The full posterior update beats prior-only, evidence-only and product forms. Prior-only gives strong image-level fidelity (VIF 1.123, Qabf 0.728, Qcb 0.597) but lower mIoU (59.09) and mAP (55.56); evidence-only raises AG (3.885) and SCD (1.696) but weakens downstream performance (mIoU 59.04, mAP 55.38); the product form is less stable (SCD 1.596); the full normalized posterior gives the best downstream result (mIoU 60.05, mAP 55.98).

• The symmetric exposure-failure prior is the strongest context prior. The form (2Y_v − 1)² reaches VIF 1.117, Qcb 0.593, mIoU 60.05 and mAP 55.98, ahead of 1 − Y_v (VIF 1.111, mIoU 58.88), Y_v (mIoU 59.19) and |2Y_v − 1| (mIoU 58.94).

• Final structural fidelity is indispensable. Removing the detail loss collapses Qabf to 0.178, Qcb to 0.332, mIoU to 41.01 and mAP to 23.83. Removing the proposal loss (mAP 56.00) or the context calibration loss (mIoU 58.90) or the keep term (mIoU 58.79) each lowers downstream scores relative to the full model, and applying saliency supervision directly to the final output raises SCD (1.735) but lowers VIF, Qabf, Qcb, mIoU (58.56) and mAP (55.15).

• Ordered P–C–D rendering matters. P–C–D scores AG 3.839, VIF 1.117, Qabf 0.723, mIoU 60.05, mAP 55.98, ahead of C–P–D (mIoU 56.30), P–D–C (56.24) and a parallel joint-gating variant (56.49); the paper reports improvements of 0.015 AG, 0.007 VIF, 0.003 Qabf, 3.56 mIoU and 0.30 mAP over the strongest alternative per metric.

• Robustness to input corruption on FMB mIoU. Across 15 corruption settings, PCDFusion ranks first in 11 and second in the remaining four, and achieves the highest mIoU in eight of nine single-modality corruption settings. Under visible dark corruption at severities 1/2/3 it reports 59.75 / 57.09 / 52.14 versus SAGE 56.72 / 53.79 / 51.02 and UPFusion 55.93 / 51.75 / 47.37; with infrared only corrupted it reports 58.06 / 57.44 / 56.54. Under joint corruption it remains best for both severity-1 settings and the severity-2 VIS-dark setting and stays above UPFusion in all six joint cases, although SAGE is better under several severe joint corruptions.

• Posterior maps behave as intended. The visualization analysis shows the posteriors refine their priors: the proposal weight strengthens reliable thermal-response regions, the context weight restricts updates to unreliable visible areas, and the detail weight localizes final detail compensation; the context weight also tracks empirical exposure-failure trends on both MSRS and FMB.

• Stated limitations. Residual blurring remains in low-luminance regions (tree branches), and minor thermal interference still transfers into fused output where infrared is invalid (glass region); the authors attribute this partly to the context posterior staying relatively high in low-luminance ranges.

Methodology in Plain English

The authors start from a simple observation: visible and infrared sensors fail in different ways, so the fusion rule should be decided locally rather than globally. For each of three rendering steps they define a yes/no decision in probabilistic terms. A physical cue — normalized infrared intensity, visible luminance, or a comparison of structural strength — provides a starting belief, the prior. A small neural network looks at the visible-anchored features plus the visible and infrared observations and outputs a correction to the odds of taking the action, the learned log-evidence. Combining prior and evidence gives a number between 0 and 1, the posterior action weight, which controls how much of the action's output is blended with the unchanged baseline.

These three decisions are stacked in a fixed order. The proposal stage adds a bounded positive amount of infrared-driven brightness into the visible luminance, capped by the remaining headroom, so candidate thermal targets are kept but not forced. The context stage then decides whether to replace the current luminance with a recovered version produced from a value prediction plus a bounded residual, aimed at regions that are too dark or saturated. The detail stage extracts a bounded, band-limited infrared residual and adds it into the luminance with a posterior weight, so low-frequency thermal background is excluded and only sharper structures come through.

The visible stream is a Swin Transformer with patch size 8, embedding dimension 192 and depth 4, kept at H/8 × W/8 resolution; the infrared stream is a lightweight convolutional encoder with base width 24 producing features at H/2, H/4, H/8 and H/16 with widths 48, 96, 192 and 192, injected into the visible tokens via cross-attention where visible tokens supply queries and infrared tokens supply keys and values. The fused hidden feature has width 48 and feeds the three rendering heads. Training uses three losses: a one-sided recall loss that pushes the proposal output toward a relaxed thermal target inside an infrared-salient mask, a calibration loss that matches the context weight to a stopped-gradient product of the exposure prior and infrared structure plus a keep term that penalizes context updates in reliable visible regions, and a gradient-magnitude loss that matches the final image's gradient magnitude to the element-wise maximum of the visible and infrared gradient magnitudes. Training runs 200 epochs with AdamW, learning rate 2×10⁻⁴, weight decay 10⁻⁴, batch size 16 and 256×256 crops with cosine decay; at inference the fused luminance is combined with visible chrominance to form the RGB output.

Why This Matters

• Research impact: The paper supplies an alternative to symmetric feature mixing in IVIF, moving the probabilistic decision from the fused image or the generation process onto individual rendering actions with an explicit prior anchor and a signed evidence correction. It also shows that ordered, dependent rendering decisions outperform both reordered and parallel gating, which is a design principle other multimodal fusion pipelines can test.
• Real-world applications:
  • Night-time or adverse-weather driving assistance, where headlight glare, saturation or darkness degrades the visible stream but thermal targets remain detectable.
  • Surveillance and security imaging, where illumination changes are frequent and thin structures such as poles or fences must survive fusion.
  • Search-and-rescue and inspection drones, where the same fused representation feeds both human observers and automatic detectors.
  • Robotics and autonomous perception stacks that rely on segmentation and detection models retrained on fused imagery.
• Industry relevance: Because the fusion model is trained on a single dataset (MSRS) and evaluated on others without fine-tuning, and because gains carry through to SegFormerB2 segmentation and YOLOv10s detection, the approach targets practical deployment where labeled fusion data is scarce — the authors explicitly train without semantic labels for the fusion objective. The code is released at https://github.com/BIT-RuiLiu/PCDFusion.git.

Future Directions

• Stronger posterior calibration in low light. The authors note the context posterior remains relatively high in low-luminance ranges, making the model over-permissive when infrared is unreliable; recalibration with stronger structure-preservation constraints is proposed.
• Explicit infrared reliability estimation. Both the limitations and conclusion call for a more direct estimate of when infrared responses should be distrusted, motivated by the glass-region interference shown in the qualitative results.
• Salvaging weak visible structure while suppressing invalid thermal signal. The paper frames the remaining hard case as preserving faint but genuine visible detail without admitting spurious thermal contributions — a tension the current prior/evidence split does not fully resolve.
• Extending the ordered posterior-rendering template beyond three stages or other modality pairs. Since the same prior–evidence–posterior update is reused at every stage, a natural open question is whether additional or differently ordered decisions improve results in other heterogeneous sensing combinations.

Target Audience

Graduate students and researchers working on multimodal image fusion, computational imaging, or Bayesian/probabilistic deep learning who want a concrete worked example of turning a probabilistic argument into network architecture and losses. It is also relevant to applied engineers building night-time or low-light perception pipelines that depend on fused infrared–visible inputs for detection and segmentation, and to readers interested in unsupervised or label-free training objectives for vision tasks.

Authors’ abstract

Infrared-visible image fusion (IVIF) aims to combine visible and thermal infrared observations for human perception and downstream vision tasks. Visible and thermal infrared observations are physically complementary but unreliable in different ways. Existing deep IVIF methods mainly improve feature interaction and reconstruction, but often lack an explicit local rule for deciding when visible appearance should dominate and when thermal responses should contribute. To address this issue, we reformulate IVIF as local Bayesian posterior rendering, where fusion is controlled by posterior action weights rather than direct modality mixing. In this paper, we propose Proposal-Context-Detail Fusion (PCDFusion), a visible-anchored framework that implements this formulation through three rendering stages. PCDFusion preserves potentially useful thermal responses, updates exposure-degraded visible regions, and renders reliable band-limited infrared structures into the final luminance. Extensive experiments on public IVIF benchmarks show that PCDFusion improves fusion quality, target recovery, and downstream detection and segmentation performance under challenging illumination. The code is available at https://github.com/BIT-RuiLiu/PCDFusion.git.