MMMamba: A Versatile Cross-Modal In Context Fusion Framework for Pan-Sharpening and Zero-Shot Image Enhancement

AI summary

Overview

• Research area: Remote sensing image fusion (pan-sharpening) and cross-modal deep learning architectures; specifically state space models (Mamba) and the in-context conditioning paradigm.
• Technical level: Advanced. The paper assumes familiarity with pan-sharpening, cross-attention, diffusion-transformer in-context conditioning, and selective state space models.
• Scope (one sentence): The paper introduces MMMamba, a Mamba-based in-context cross-modal fusion framework that fuses panchromatic (PAN) and multispectral (MS) satellite images for pan-sharpening and, without retraining, for zero-shot MS image super-resolution.

What This Paper Is About

Satellite sensors cannot directly capture multispectral images that have both fine spatial detail and rich spectral information, so most satellites carry two sensors: a high-resolution panchromatic (PAN) image and a lower-resolution multispectral (MS) image. Pan-sharpening fuses them into a single high-resolution multispectral (HRMS) image. The paper argues that existing CNN methods use static channel concatenation and that Transformer cross-attention methods smooth out high-frequency PAN detail and pass information in only one direction, and it proposes a Mamba-based framework that instead concatenates PAN and MS tokens into one unified sequence for joint processing at linear complexity.

Key Contributions

1. MMMamba framework: A cross-modal in-context fusion framework for pan-sharpening built on the Mamba architecture, giving linear computational complexity and bidirectional information flow between the PAN and MS modalities, plus zero-shot generalization to image super-resolution.
2. First in-context conditioning in pan-sharpening: The authors state they are the first to explore the in-context conditioning paradigm in pan-sharpening, enabling joint interaction among all tokens from both modalities.
3. Multimodal Interleaved (MI) scanning mechanism: A scanning design that places spatially corresponding PAN and MS tokens adjacent to one another and scans them in four directions to exploit complementary cross-modal cues.
4. Empirical validation: Experiments on multiple benchmarks and tasks (reduced-resolution pan-sharpening, full-resolution real-world scenes, and zero-shot super-resolution) showing consistent gains over existing state-of-the-art methods.

Main Findings

• Reduced-resolution pan-sharpening gains: On the three benchmark datasets, MMMamba reaches PSNR of 42.3120 on WorldView-II, 47.9932 on GaoFen2, and 31.2311 on WorldView-III, with SSIM of 0.9733, 0.9902, and 0.9305, and SAM of 0.0209, 0.0098, and 0.0687 respectively. The paper reports PSNR gains over CFLIHPs of 0.40 dB on WV2 and 0.61 dB on GF2.
• Full-resolution evaluation: On the full-resolution GF2 dataset (no downsampling, no-reference metrics), MMMamba achieves D_lambda 0.0656, D_S 0.1113, and QNR 0.8312, which the authors describe as outperforming other approaches across all three metrics. Note that SFINet++ has a higher QNR (0.8471) and WaveletNet a higher QNR (0.8327) than MMMamba in that same table.
• Zero-shot super-resolution: Trained only on pan-sharpening and run by dropping the PAN modality, MMMamba achieves PSNR 36.4892, SSIM 0.9114, SAM 0.0299, and ERGAS 1.5515 on WV2, exceeding Bicubic (PSNR 34.0869), SFINet++ (33.3047), and Pan-Mamba (30.5913).
• Mamba vs. Transformer backbone: Replacing the Mamba operator with a computationally matched self-attention block drops PSNR from 42.3120 to 41.3995.
• In-context fusion vs. channel concatenation: Replacing in-context fusion with naive channel-wise concatenation drops PSNR to 41.2898.
• Interleaving is critical: Replacing interleaved token arrangement with sequential concatenation causes the largest degradation, with PSNR falling to 36.4702 (SSIM 0.9107, SAM 0.0302, ERGAS 1.5550).
• Multi-directional scanning helps: Reducing four-direction scanning to a single unidirectional scan lowers PSNR to 42.0965 with negligible computational change.
• Local windows help: Replacing the local window scan with a global scan lowers PSNR to 42.1998.
• Efficiency: MMMamba uses 5.0616 G FLOPs and 0.2453 M parameters, measured on PAN images of 128 x 128 and MS images of 32 x 32 on a single Nvidia V100 GPU. This is fewer parameters than SRPPNN (1.7114 M), FAME (0.5766 M), WaveletNet (1.3230 M), and Pan-Mamba (0.1827 M is lower; MMMamba is higher than Pan-Mamba and CFLIHPs at 0.1314 M).

Methodology in Plain English

The pipeline takes an upsampled low-resolution MS image and a PAN image and passes each through its own gated convolutional encoder to produce shallow features. Each feature map is normalized, linearly projected, passed through a depthwise convolution, and activated with SiLU. The features are then tokenized into non-overlapping patches.

The distinctive step is the Multimodal Interleaved (MI) scanning. Instead of lining up all MS tokens and then all PAN tokens, the method interleaves patches from the two modalities so that corresponding spatial positions sit next to each other in the sequence. This is done in four directions ("left-to-right and up-to-down", "up-to-down and left-to-right", "right-to-left and down-to-up", and "down-to-up and right-to-left"); the resulting sequences are concatenated, then split into two parts that are scanned alternately window by window across the two modalities. The four directional outputs are summed. The MI-SSM outputs are combined with the SiLU-activated normalized features through element-wise multiplication and summation, then linearly projected and reshaped back to image form. A convolutional decoder produces the final MS feature, which is added residually to the upsampled MS input to yield the HRMS image. Training uses an L1 loss against ground truth.

For zero-shot super-resolution, the authors simply omit the PAN input; because the fusion is done by concatenating tokens rather than by a dedicated cross-attention module, the model still runs. Baseline deep models that require two inputs were adapted by feeding the MS image into both the PAN and MS encoders.

Implementation details: PyTorch, one Nvidia V100 GPU, Adam optimizer, gradient clipping norm 4.0, learning rate initialized at 5 x 10^-4 with cosine decay to 5 x 10^-8, 200 epochs on WorldView-II and 500 epochs on GaoFen2 and WorldView-III. Test data was generated at reduced resolution following the Wald protocol, since ground truth at full resolution was not available. Evaluation uses PSNR, SSIM, SAM, and ERGAS as full-reference metrics, and D_S, D_lambda, and QNR as no-reference metrics.

Why This Matters

• Research impact: The paper transfers the in-context conditioning idea from Multimodal Diffusion Transformers into remote sensing image fusion, and shows that Mamba can serve as the efficient backbone for that paradigm. It also demonstrates a model trained on one task (pan-sharpening) transferring to another (super-resolution) without retraining, which is an unusual property in this literature and challenges the single-task design of prior Mamba-based fusion work.
• Real-world applications (from the paper's framing):
  • Agriculture, where high-resolution multispectral imagery supports crop monitoring.
  • Urban planning, where fine spatial detail is needed to resolve built environments.
  • Environmental monitoring, where spectral fidelity matters for detecting land and water change.
  • Satellite image products that mix pan-sharpened and multispectral-only inputs, where a single model can handle both fusion and upsampling without separate pipelines.
• Industry relevance: Satellite operators and geospatial data providers must choose among sensors and resolution levels; a compact model (0.2453 M parameters, 5.0616 G FLOPs) that handles both two-input fusion and one-input super-resolution reduces deployment and retraining cost. The code is released at https://github.com/Gracewangyy/MMMamba.

Future Directions

• Generalization beyond the tested satellites: The paper evaluates on WorldView-II, GaoFen2, and WorldView-III; whether the MI scanning design transfers to other sensors, spectral channel counts, or resolution ratios other than the s = 4 setting used here is not reported.
• Extension to other modalities and tasks: The zero-shot result is demonstrated for MS super-resolution only. Whether dropping other modalities supports additional fusion or restoration tasks is an open question the paper does not address.
• Recovering the full-resolution no-reference advantage: In Table 2, MMMamba has QNR 0.8312 while SFINet++ reports 0.8471 and WaveletNet reports 0.8327, so improving no-reference quality on real-world full-resolution scenes remains an open target even though D_lambda and D_S are best.
• Dataset scale and training cost: The paper does not report dataset sizes (number of images or scenes), training times, or memory usage, which limits assessment of how well the method scales to larger archives.

Target Audience

Graduate students and researchers in remote sensing, image fusion, and efficient vision architectures; engineers building satellite image processing pipelines who need a compact model covering both pan-sharpening and super-resolution; and readers interested in state space models or in-context multimodal conditioning applied outside of language and generative image models.

Authors’ abstract

Pan-sharpening aims to generate high-resolution multispectral (HRMS) images by integrating a high-resolution panchromatic (PAN) image with its corresponding low-resolution multispectral (MS) image. To achieve effective fusion, it is crucial to fully exploit the complementary information between the two modalities. Traditional CNN-based methods typically rely on channel-wise concatenation with fixed convolutional operators, which limits their adaptability to diverse spatial and spectral variations. While cross-attention mechanisms enable global interactions, they are computationally inefficient and may dilute fine-grained correspondences, making it difficult to capture complex semantic relationships. Recent advances in the Multimodal Diffusion Transformer (MMDiT) architecture have demonstrated impressive success in image generation and editing tasks. Unlike cross-attention, MMDiT employs in-context conditioning to facilitate more direct and efficient cross-modal information exchange. In this paper, we propose MMMamba, a cross-modal in-context fusion framework for pan-sharpening, with the flexibility to support image super-resolution in a zero-shot manner. Built upon the Mamba architecture, our design ensures linear computational complexity while maintaining strong cross-modal interaction capacity. Furthermore, we introduce a novel multimodal interleaved (MI) scanning mechanism that facilitates effective information exchange between the PAN and MS modalities. Extensive experiments demonstrate the superior performance of our method compared to existing state-of-the-art (SOTA) techniques across multiple tasks and benchmarks.