Text-Guided Channel Perturbation and Pretrained Knowledge Integration for Unified Multi-Modality Image Fusion

AI summary

Overview

Research area: Computer vision — multi-modality image fusion (MMIF), specifically unified models that handle infrared-visible, medical, and other fusion tasks with a single set of weights.

Technical level: Advanced. The paper assumes familiarity with autoencoder architectures, Transformer blocks, channel attention, CLIP text encoders, and image-fusion evaluation metrics.

Scope: The authors propose UP-Fusion, a unified fusion framework combining a Semantic-Aware Channel Pruning Module (SCPM), a Geometric Affine Modulation Module (GAM), and a Text-Guided Channel Perturbation Module (TCPM), trained once on LLVIP and evaluated on infrared-visible fusion, medical fusion, and two downstream tasks.

What This Paper Is About

Multi-modality image fusion combines complementary information from different sensors — for example, thermal radiation from infrared plus fine texture from visible light, or CT plus MRI in medicine — into one more informative image. Existing unified models that share parameters across modalities generalize well but fuse poorly, while models with separate encoders per modality fuse better but generalize worse to new modality combinations. This paper tries to get both: good fusion quality and strong cross-task generalization from a single shared set of weights.

Key Contributions

1. A unified MMIF framework (UP-Fusion) built on text-guided channel perturbation and pretrained knowledge, intended to reduce modal redundancy while improving unified modeling and generalization across cross-modal features.
2. The Semantic-Aware Channel Pruning Module (SCPM), which fuses channel attention with global semantic guidance from a pretrained ConvNeXt model to filter redundant channels and strengthen salient features, plus the Geometric Affine Modulation Module (GAM), which applies affine transformations to fusion representations based on modality-specific features.
3. Extensive experiments reported as outperforming both task-specific and unified fusion models on infrared-visible and medical image fusion, with additional gains on downstream applications.

Main Findings

• Single weight set across tasks: The model is trained only on the LLVIP dataset and uses one weight set for all MMIF tasks, yet is evaluated on MSRS, LLVIP, M3FD, Harvard Medical School brain images (CT-MRI, PET-MRI, SPECT-MRI), and a GFP-PC database.
• Infrared-visible results (Table 1): UP-Fusion reaches Q_NCIE 0.8167 / Q_P 0.5671 / VIF 0.4587 / SSIM 0.5074 / Q^AB/F 0.6859 on MSRS; 0.8092 / 0.5042 / 0.3817 / 0.4636 / 0.6864 on LLVIP; and 0.8119 / 0.5190 / 0.4582 / 0.5163 / 0.6275 on M3FD — reported as the optimum in multiple metrics on all three datasets.
• Medical fusion results (Table 1 and Table 2): On the Harvard set, UP-Fusion scores Q_NCIE 0.8074 / Q_P 0.5665 / VIF 0.3190 / SSIM 0.3639 / Q^AB/F 0.6814, described as ranking first in four metrics and second in one against 10 comparison methods. On GFP-PC it scores 0.8170 / 0.4826 / 0.3193 / 0.3339 / 0.5914.
• Qualitative cross-task behavior: Although trained only on infrared-visible data, the method is reported to accurately extract bone structures in CT images and detailed textures in MRI, and to outperform medical-specific fusion methods in color fidelity. In contrast, SAGE and CoCoNet lose detail on medical fusion, and GIFNet and CrossFuse introduce excessive black background regions.
• Ablation (Table 3, medical task): Removing SCPM drops SSIM to 0.2645; removing GAM gives 0.5488 Q_P / 0.3478 SSIM; removing TCPM gives 0.5221 Q_P / 0.2824 SSIM; removing channel attention inside TCPM gives 0.5057 Q_P / 0.2632 SSIM; removing ConvNeXt gives 0.3018 VIF / 0.2807 SSIM; removing channel attention inside SCPM gives 0.5441 Q_P / 0.3218 SSIM. Full UP-Fusion is 0.8074 / 0.5665 / 0.3190 / 0.3639.
• Module roles in ablation: Without GAM, black background information in SPECT is incorrectly fused; without ConvNeXt, cross-task performance drops because the channel attention in SCPM may select channels incorrectly on unseen tasks; removing channel attention blurs textures.
• Downstream segmentation (Table 4, MSRS, BANet): UP-Fusion reaches mIoU 78.28, ahead of TDFusion (78.14), EMMA (77.77), SAGE (75.94), GIFNet (74.63), CrossFuse (70.74), CoCoNet (70.5), and TIMFusion (66.67). It leads on several categories, including Background 98.56, Car 90.49, Bike 71.94, Cuardrail 85.63, Color cone 66.1, and Bump 80.22.
• Downstream detection (Table 5, M3FD, YOLOv7): UP-Fusion leads on mAP@0.5 (0.841) and mAP@[0.5:0.95] (0.541), compared with TDFusion (0.836 / 0.535) and SAGE (0.836 / 0.532).

Methodology in Plain English

The system is an autoencoder built from Transformer blocks: a 4-layer encoder and 4-layer decoder, with the number of Transformer blocks increasing as [4, 6, 6, 8] and attention heads as [1, 2, 4, 8], so spatial resolution shrinks while channel capacity grows.

Three ideas drive the design:

1. Clean up the channels first (SCPM). Because multi-modal data is redundant, the model computes a channel-importance score using a Squeeze-and-Excitation-style channel attention mechanism, and separately extracts semantic guidance from a pretrained ConvNeXt backbone. The two are combined as ω_F = ω_C + α · σ(ω_S), where α is learnable and σ is Sigmoid. The top 70% of channels are kept and a 1×1 convolution restores the original dimensionality. The output is split into two subsets, one per modality, for the next stage.

2. Modulate fusion geometrically rather than injecting modality directly (GAM). Global average pooling gives per-modality summary vectors, which pass through a two-layer 1×1 convolution network to produce a scale γ and offset β. The fused features are then adjusted as F_O^M = Fuse^M · (1 + γ) + β, letting the fused representation follow the global geometric character of each modality without letting the encoder overfit to one modality.

3. Perturb channels during decoding (TCPM). Multi-modality features are concatenated and ranked by channel attention, keeping the top 50%. A 1×1 convolution doubles the channel count to add redundancy and expressiveness. CLIP-encoded text features go through a two-layer linear mapping to produce weights that generate a channel rearrangement index, guiding semantically relevant rearrangement. The perturbed features serve as queries against the original features as keys and values in a self-attention step, followed by a feed-forward network.

Training uses only two losses — a gradient loss that pushes the fused gradient toward the maximum of the source gradients, and an L1 loss toward each source image — summed as the total loss. Training runs 100 epochs on 192×192 random crops, batch size 2, learning rate 0.0001 cosine-decayed to 0.00001, with the Adam optimizer, in PyTorch 2.1.1 on four GeForce RTX 3090 GPUs. Evaluation uses five metrics: Q_NCIE, Q_P, VIF, SSIM, and Q^AB/F. The paper does not report dataset sizes for MSRS, LLVIP, M3FD, Harvard, or GFP-PC.

Why This Matters

Impact on research: The paper reframes the unified-versus-specialized tradeoff in MMIF as a problem of modality dependence in the channel distribution, rather than only a question of encoder sharing or learning strategy. It also brings text guidance (via CLIP) and channel perturbation into fusion, extending ideas previously explored in other vision settings and pointing toward architectures whose generalization comes from explicitly erasing modal markers.

Real-world applications:

• Infrared-visible fusion for surveillance, automotive night vision, and other machine-vision systems operating under complex lighting.
• Medical imaging, where CT, MRI, PET, and SPECT information is combined for a more complete view of pathological features.
• Fluorescence microscopy, via the GFP-PC evaluation setting.
• Pretrained perception pipelines, since improved fused images feed object detection and semantic segmentation.

Industry relevance: A single set of weights covering many fusion tasks reduces the cost of maintaining and deploying per-task models, which matters for embedded and automotive perception stacks where memory and retraining budgets are constrained. The reported downstream gains under identical experimental settings (BANet for segmentation, YOLOv7 for detection) indicate the fused outputs are usable by standard downstream models rather than needing bespoke training.

Future Directions

• The paper trains exclusively on LLVIP and tests zero-shot on medical and fluorescence data; how the approach behaves when some target-modality data is available for adaptation is not explored.
• The text guidance in TCPM is described only generically as CLIP-encoded text features; what text is used, and how sensitive the method is to its wording, is not reported.
• The paper notes that existing unified approaches rely on learning strategies such as continual learning or meta-learning; whether combining those with UP-Fusion's architectural approach would add further gains is an open question.
• The system was trained at 192×192 crops on four RTX 3090 GPUs; the paper reports no inference cost, parameter count, or runtime comparison, which would be needed to judge deployment practicality.

Target Audience

Researchers and graduate students working on image fusion, multi-modal perception, and unified architectures for computer vision, as well as practitioners who need one model to handle several fusion tasks without per-task retraining. Readers should be comfortable with autoencoder and Transformer terminology, channel attention, CLIP-style text encoders, and standard fusion quality metrics.

Authors’ abstract

Multi-modality image fusion enhances scene perception by combining complementary information. Unified models aim to share parameters across modalities for multi-modality image fusion, but large modality differences often cause gradient conflicts, limiting performance. Some methods introduce modality-specific encoders to enhance feature perception and improve fusion quality. However, this strategy reduces generalisation across different fusion tasks. To overcome this…