Self-supervised Multiplex Consensus Mamba for General Image Fusion

AI summary

Overview

Research area: Computer vision, specifically general image fusion (infrared-visible, medical, multi-focus, and multi-exposure fusion) using state space models.

Technical level: Advanced. The paper assumes familiarity with State Space Models (Mamba), Mixture of Experts, Fourier transforms, contrastive learning, and standard image fusion evaluation metrics.

Scope: The paper introduces SMC-Mamba, a single self-supervised framework built on Mamba that fuses image pairs across four different fusion tasks using three components — a Modality-Agnostic Feature Enhancement module, a Multiplex Consensus Cross-modal Mamba module, and a Bi-level Self-supervised Contrastive Learning loss.

What This Paper Is About

Most deep image fusion methods are built for one task at a time, so a model trained on infrared-visible fusion does not transfer well to medical or multi-focus fusion. Existing deep networks also tend to favor low-frequency content, which causes them to lose fine textures and edges, and prior regularization strategies can be inefficient at recovering them.

The goal is a single general framework that dynamically adapts to different modality pairs, preserves high-frequency detail without adding computational overhead, and produces fused images that also help downstream tasks such as object detection and semantic segmentation.

Key Contributions

1. SMC-Mamba framework. The authors propose a Self-supervised Multiplex Consensus Mamba for general image fusion that dynamically and efficiently integrates complementary information from various modalities across different fusion tasks (IVIF, MDIF, MFIF, MEIF).

2. Multiplex Consensus Cross-modal Mamba (MCCM). A module that combines multiple cross-modal Mamba experts (the paper sets N = 4) under a unified gating framework, promoting diverse feature preferences and fusion strategies across experts while driving the activated experts toward a unified consensus representation.

3. Bi-level Self-supervised Contrastive Learning Loss (BSCL). A regularization loss that preserves high-frequency information at both feature and pixel levels — pulling fused high-frequency components toward the input high-frequency components and pushing them away from low-frequency components — without increasing model complexity, while also improving downstream visual task performance.

4. Cross-modal scanning. A scanning mechanism embedded in each MCCM expert that exploits long-range cross-modal dependencies, with spatial scanning performing forward and reverse passes between modalities and channel scanning alternating across modalities.

Main Findings

• IVIF results on MSRS: The proposed method achieves MI 4.490, SF 12.211, AG 4.054, CC 0.699, SCD 1.664, VIF 0.991, Q_abf 0.658, and MS_SSIM 0.522, the best score on each of these metrics among the compared methods.

• IVIF results on RoadScene: The proposed method reports MI 3.772, SF 17.971, AG 6.866, CC 0.643, SCD 1.733, VIF 0.642, Q_abf 0.557, and MS_SSIM 0.547, again best across the listed metrics.

• IVIF results on M3FD: The proposed method reports MI 4.280, SF 19.495, AG 6.378, CC 0.561, SCD 1.791, VIF 0.972, Q_abf 0.632, and MS_SSIM 0.507, best across the listed metrics.

• MFIF results on Lytro: The proposed method reports MI 7.081, SF 23.785, AG 8.191, CC 0.989, SCD 0.787, VIF 1.339, N_abf 0.007, and MS_SSIM 0.899.

• MFIF results on MFI-WHU: The proposed method reports MI 6.890, SF 35.669, AG 10.929, CC 0.985, SCD 0.972, VIF 1.344, N_abf 0.070, and MS_SSIM 0.948. The paper states the method outperforms existing approaches across nearly all metrics and datasets.

• Downstream semantic segmentation: With DeepLabV3+ on MSRS, the proposed method reaches mIoU 79.3 (Background 98.7, Car 90.0, Person 73.7, Bike 72.6, Curve 65.6, Barrier 75.0), compared with LFDT-Fusion at 78.9 and MLFuse at 78.8.

• Efficiency versus alternatives: The full model uses 0.149 M parameters, 46.105 G FLOPs, and 288.545 ms inference time. Replacing Mamba with convolution gives 0.325 M parameters, 78.843 G FLOPs, and 430.392 ms; replacing it with window attention gives 0.392 M, 58.313 G, and 792.461 ms; self-attention gives 0.240 M, 60.747 G, and 1271.691 ms — all lower-performing than the proposed configuration.

• Module ablation: Removing the MAFE module drops MI to 2.384 with 0.041 M parameters, 14.260 G FLOPs, and 226.355 ms. Removing the MCCM module drops MI to 2.202 with 0.125 M parameters, 38.606 G FLOPs, and 164.867 ms.

• Loss ablation: Removing both contrastive losses (L_fcl and L_pcl) lowers MI to 3.721; removing L_mccm lowers MI to 3.466; removing L_div gives MI 3.601; removing L_cons gives 3.702; removing L_wb gives 3.840; removing only L_fcl gives 3.914 and removing only L_pcl gives 3.870.

• Scanning ablation: Removing cross-modal scanning lowers MI to 3.965 and SCD to 1.470; removing spatial-channel scanning gives MI 4.106; removing frequency-rotational scanning gives MI 4.350; switching bidirectional to single-direction scanning gives MI 4.270.

• Visual comparisons: The paper reports that on MSRS only the proposed method clearly highlights pedestrian targets within the red box, and on MFI-WHU it preserves fine-grained textures such as sharp railings and clear flag lines while maintaining accurate color fidelity.

• Not reported in the provided content: Quantitative tables for the medical image fusion (CT-MRI, PET-MRI, SPECT-MRI on the Harvard medical dataset) and multi-exposure fusion (MEF dataset) tasks are referenced as part of the experiments but the tables themselves are not included in the supplied text. The paper mentions using MSRS and M3FD for downstream detection evaluation, but detection numbers are also not shown in the provided content.

Methodology in Plain English

The system takes two source images ("modalities") and processes them through three stages.

Stage one — feature enhancement (MAFE). Each source image is converted into shallow features with a 3×3 convolution and layer normalization. Two branches then work in parallel. A local branch splits features into patches, applies 3×3 depth-wise convolution, and uses a gating unit to adaptively keep fine-grained spatial detail. A global branch uses two Mamba-based scans: a spatial-channel scan and a frequency-rotational scan. For the frequency path, the features are converted into the Fourier domain, split into amplitude and phase, processed with depth-wise convolution and SiLU activation, scanned, and converted back with an inverse Fourier transform. The local and global outputs are concatenated.

Stage two — cross-modal fusion (MCCM). Multiple Mamba "experts" each perform cross-modal fusion on the combined features. A gating network built from global average pooling, global max pooling, and a learnable noise term decides which experts matter, activating only the top-2 experts at inference (top-2 of N = 4 during training all experts are used with computed weights). The output is a weighted sum of expert outputs. Three training losses keep the experts healthy: a workload balancing loss based on the coefficient of variation to prevent gating collapse, an expert diversity loss based on cosine similarity between expert outputs to keep experts specialized, and a consensus loss penalizing deviation from a weighted-average consensus feature. These are combined with a time-decayed weight that favors diversity early and consensus later.

Stage three — self-supervised contrastive regularization (BSCL). Using a Haar wavelet lifting scheme, the paper decomposes both features and images into high- and low-frequency parts through channel splitting, prediction, and update steps. Contrastive ratios at feature level (L_fcl) and pixel level (L_pcl) pull fused high-frequency components toward the source high-frequency components and push them away from the low-frequency components. The overall objective combines L_fcl, L_pcl, L_mccm, an SSIM loss, and an intensity loss with empirically set weights of 0.8, 0.4, 1, 1, and 1.

Training setup. PyTorch on a single NVIDIA RTX 3090 GPU, ADAM optimizer with β = 0.9, batch size 1, initial learning rate 2 × 10⁻⁴ halved every 1000 iterations via cosine annealing. A 3×3 convolution and layer normalization produce the shallow features.

Why This Matters

Impact on research. The work argues that a single general fusion architecture can serve multiple fusion tasks, and it positions Mamba as a middle ground between CNNs (limited receptive field) and Transformers (quadratic cost). It also introduces contrastive regularization as a way to recover high-frequency detail without adding parameters, addressing a documented bias toward low-frequency content in deep networks.

Real-world applications:

• Infrared-visible fusion for surveillance and autonomous driving, where thermal pedestrians must remain visible in fused output.
• Medical imaging, combining CT-MRI, PET-MRI, and SPECT-MRI to support diagnosis with complementary structural and functional information.
• Multi-focus fusion for microscopy and photography, where all-in-focus images are needed from differently focused shots.
• Multi-exposure fusion for photography, producing well-exposed images from under- and over-exposed inputs.

Industry relevance. The ablation reports 0.149 M parameters and 288.545 ms inference time, which is far leaner than the self-attention variant at 0.240 M parameters and 1271.691 ms, making the approach relevant where deployment cost matters. Improved mIoU (79.3 on MSRS with DeepLabV3+) indicates the fused images transfer as usable inputs to downstream perception pipelines.

Future Directions

• Report the missing quantitative results. The medical (CT-MRI, PET-MRI, SPECT-MRI) and multi-exposure results, plus the downstream detection evaluations mentioned for MSRS and M3FD, are not present in the provided content and would need to be shown to support the claim of outperforming state of the art across all four fusion tasks.
• Generalize beyond pairs. The framework is formulated for modality index k ∈ {1, 2}; extending the cross-modal scanning and expert gating to more than two inputs is an open question.
• Reduce the gap to ablated variants on specific cases. The full model uses more parameters and FLOPs than the MAFE-removed variant (0.149 M vs 0.041 M; 46.105 G vs 14.260 G), so further efficiency work may be worthwhile.
• Justify expert configuration. Only N = 4 experts with top-2 activation is evaluated; how the diversity/consensus tradeoff scales with different expert counts is not explored in the provided content.

Target Audience

Researchers and graduate students working on image fusion, state space models, or multi-modal representation learning. Practitioners building perception pipelines for surveillance, autonomous driving, or medical imaging who need a single fusion model that handles multiple modality pairs will also benefit, as will those evaluating mixture-of-experts designs, cross-modal scanning, or contrastive regularization for image restoration tasks.

Authors’ abstract

Image fusion integrates complementary information from different modalities to generate high-quality fused images, thereby enhancing downstream tasks such as object detection and semantic segmentation. Unlike task-specific techniques that primarily focus on consolidating inter-modal information, general image fusion needs to address a wide range of tasks while improving performance without increasing complexity. To achieve this, we propose SMC-Mamba, a Self-supervised Multiplex Consensus Mamba framework for general image fusion. Specifically, the Modality-Agnostic Feature Enhancement (MAFE) module preserves fine details through adaptive gating and enhances global representations via spatial-channel and frequency-rotational scanning. The Multiplex Consensus Cross-modal Mamba (MCCM) module enables dynamic collaboration among experts, reaching a consensus to efficiently integrate complementary information from multiple modalities. The cross-modal scanning within MCCM further strengthens feature interactions across modalities, facilitating seamless integration of critical information from both sources. Additionally, we introduce a Bi-level Self-supervised Contrastive Learning Loss (BSCL), which preserves high-frequency information without increasing computational overhead while simultaneously boosting performance in downstream tasks. Extensive experiments demonstrate that our approach outperforms state-of-the-art (SOTA) image fusion algorithms in tasks such as infrared-visible, medical, multi-focus, and multi-exposure fusion, as well as downstream visual tasks.