3M-TI: High-Quality Mobile Thermal Imaging via Calibration-free Multi-Camera Cross-Modal Diffusion

AI summary

Overview

Research area: Computational thermal imaging, cross-modal (RGB–thermal) image fusion, and diffusion-based image super-resolution for mobile platforms.

Technical level: Advanced. The paper builds on latent diffusion (SD-Turbo), variational autoencoders, transformer self-attention, and LoRA fine-tuning, and assumes familiarity with super-resolution and thermal infrared sensing.

Scope: The paper proposes 3M-TI, a calibration-free multi-camera cross-modal diffusion framework that reconstructs high-resolution thermal images from low-resolution thermal observations guided by uncalibrated RGB reference images, and validates it on public benchmarks and a real smartphone-based thermal camera system.

What This Paper Is About

Thermal cameras on mobile devices are small and cheap, which forces low sensor resolution and produces blurry, detail-poor images. Existing fixes either try to super-resolve a thermal image alone (which has too little information to recover fine structure) or use a high-resolution RGB image as a guide (which normally requires laborious pixel-level calibration between the two cameras and breaks down when the cameras are misaligned). The goal of this paper is to get the detail benefits of RGB guidance while removing the calibration requirement entirely, so the method works on handheld multi-camera devices with parallax and unsynchronized captures.

Key Contributions

1. Calibration-free fusion. A cross-modal self-attention module (CSM) performs fully automatic alignment and fusion of uncalibrated RGB–thermal pairs inside the latent space of a VAE, without any pixel-level image registration.
2. Misalignment augmentation. A camera-pose augmentation strategy generates intentionally misaligned RGB counterparts (translation, scaling, rotation, perspective warping) so the CSM learns to handle multi-camera parallax and unsynchronized captures rather than overfitting to a calibrated setup.
3. Cross-modal latent diffusion. The framework integrates one-step latent-space diffusion (built on SD-Turbo) with LoRA fine-tuning of both the UNet and the VAE decoder, using pretrained generative priors to suppress fusion artifacts, enhance detail realism, and compensate for limited thermal training data.
4. Practical validation. The method is evaluated on public datasets and on a real smartphone-based multi-camera thermal system, with downstream object detection and semantic segmentation experiments to show practical value.

Main Findings

• Best perceptual quality on public datasets. On the combined test set, 3M-TI with augmentation records LPIPS 0.1787, MANIQA 0.4443, and MUSIQ 36.66, which the paper reports as the best among all compared methods. For comparison, SeeSR records LPIPS 0.1828, MANIQA 0.4278, MUSIQ 35.22; OSEDiff records LPIPS 0.2113, MANIQA 0.4014, MUSIQ 36.30; DifIISR records LPIPS 0.3484, MANIQA 0.4214, MUSIQ 36.74.
• Fidelity metrics are mixed. 3M-TI with augmentation records PSNR 30.09 and SSIM 0.8610. The paper states this shows better structural preservation than OSEDiff, SeeSR, DifIISR, SwinFuSR, and SwinPaste. In the same table, CoRPLE reports the highest PSNR (30.47) and SSIM (0.8642), and CoReFusion with augmentation reports PSNR 30.30 and SSIM 0.8634.
• Diffusion priors win on perception, not fidelity. The reference-based LPIPS results show 3M-TI, OSEDiff, and SeeSR outperforming the non-diffusion methods (CoReFusion, CoRPLE, SwinFuSR, SwinPaste). Non-diffusion methods tend to reach high PSNR/SSIM but produce overly smooth reconstructions that miss high-frequency structures, while OSEDiff and SeeSR synthesize high-frequency content that is often inconsistent with ground truth.
• Augmentation helps 3M-TI much more than baselines. Misalignment augmentation improves 3M-TI notably on perceptual metrics, while other methods see only marginal gains — evidence that the model is actually learning cross-modal correspondence under uncalibrated conditions.
• Strong real-world smartphone performance. On the authors' captured smartphone dataset (300 pairs across 100 scenes, including 200 nighttime captures), 3M-TI scores MUSIQ 30.62 and MANIQA 0.3589, the highest reported, ahead of SeeSR (30.15 / 0.3454), OSEDiff (29.85 / 0.3285), SwinPaste (26.17 / 0.2749), SwinFuSR (25.99 / 0.2754), and CoReFusion (25.74 / 0.2701).
• Robustness to misalignment and degraded references. Qualitative results show 3M-TI recovering the circular shape of bicycle wheels despite significant positional misalignment from unsynchronization, and recovering a railing structure from an RGB reference heavily corrupted by light flare.
• Downstream detection gains. Using Grounded-SAM zero-shot with identical prompts, 3M-TI thermal images achieve Precision 0.4565, Recall 0.5455, F1-score 0.4724, and IoU 0.3427, surpassing the reference RGB results (0.4322 / 0.5708 / 0.4643 / 0.3359) on F1 and IoU and closely approaching the ground-truth thermal scores (0.4582 / 0.5793 / 0.4887 / 0.3494). SeeSR-scored thermal images reach 0.3832 / 0.4637 / 0.3849 / 0.3022, and SwinPaste 0.1800 / 0.2109 / 0.1765 / 0.1941.
• Downstream segmentation gains. 3M-TI produces more accurate and coherent segmentation maps than other methods; in one low-light example it outperforms the RGB reference (e.g., wheels and the upper automobile), and in another it segments trees more precisely than RGB while missing a small automobile in the bottom-left.
• Ablation results. Removing the RGB reference drops MUSIQ to 32.85 and worsens LPIPS to 0.2106. Removing augmentation drops MUSIQ to 34.94 (LPIPS 0.1917). Removing the skip connection drops PSNR to 29.86 and MUSIQ to 36.58. Removing the RAM prompt barely affects reference metrics but lowers MUSIQ to 36.20. CSM outperforms plain self-attention (MUSIQ 34.47), feature concatenation (32.12), and standard cross-attention (34.32).

Methodology in Plain English

The system takes two inputs: a low-resolution thermal image and a high-resolution RGB photo of the same scene that has not been aligned to the thermal camera.

1. Encode into latent space. Both images are passed through a frozen VAE encoder, which compresses them into compact latent representations. Correspondence is established in this latent space rather than at the pixel level, which makes the alignment problem much more forgiving.
2. Fuse with a new attention module. The core innovation, the cross-modal self-attention module (CSM), replaces the existing self-attention layers inside a diffusion UNet. The RGB and thermal latents are concatenated and reshaped so that every pixel of both modalities becomes a token in a single sequence; attention then computes dependencies across all RGB and thermal tokens at once. This lets the network learn both RGB-to-thermal guidance and thermal-to-thermal structural relationships. Non-transformer blocks simply process the two modalities independently by folding them into the batch dimension. The module adds no new parameters.
3. Train for misalignment. Because existing RGB–thermal datasets are small and strictly pixel-aligned, the authors deliberately apply translation, scaling, rotation, and perspective warping to the RGB images during training. This teaches the attention module to tolerate the parallax and timing offsets of real handheld multi-camera setups.
4. Add structural guardrails. Skip connections from four encoder downsampling blocks (through 1×1 zero-initialized convolutions) feed into the corresponding decoder blocks to preserve structure and reduce geometric distortion. A zero-initialized skip also merges the refined thermal latent with the initial thermal latent.
5. Guide with text, run fast. Text prompts are generated from the RGB images using the Recognize Anything Model (RAM), since reliable semantics cannot easily be extracted from a 64×64 thermal image. One-step latent diffusion (SD-Turbo) keeps inference efficient, and LoRA adapters (rank 16 for the UNet, rank 4 for the VAE decoder) fine-tune the model with a combined L2 and LPIPS loss.
6. Test on real hardware. The authors built a system pairing a HIKVISION P09 thermal module (under 100 US Dollars, native 96×96 resized to 64×64, 50°×50° FOV, 12 µm pixel pitch) with the primary camera of a Xiaomi 15 (OV50H sensor, 4096×3072, 23 mm equivalent focal length, 74°×59° FOV) over a Type-C interface, and collected 300 real pairs across 100 scenes.

Training details: Adam optimizer, learning rate 2×10⁻⁵, batch size 4, a single NVIDIA A800 (80 GB) GPU, approximately 4 hours and 8000 iterations. The combined training set of 10,922 pairs came from IRVI (3,200), LLVIP (3,200), M³FD (3,822), and the PBVS 2025 TISR Challenge Track 2 training split (700). The test set of 1,176 pairs came from IRVI test (300), LLVIP (300), M³FD (376), and PBVS 2025 validation (200). Thermal images were center-cropped, resized to 64×64, and given added Gaussian noise; RGB images were resized to 512×512.

Why This Matters

Impact on research. The paper reframes RGB-guided thermal super-resolution as a latent-space alignment problem rather than a pixel-registration problem. If cross-modal correspondence can be learned by attention instead of calibrated hardware, the strict-alignment assumption baked into most existing RGB–thermal datasets and methods becomes optional — which matters given that the paper notes thermal data is scarce and existing datasets are small and homogeneous.

Real-world applications:

• Autonomous driving and advanced driver assistance, where thermal sensing works in darkness, fog, and smoke and sharper thermal images improve pedestrian and vehicle detection.
• Robotic navigation and situational awareness in low-light or obscured environments, where thermal perception complements visible-light cameras.
• Nighttime surveillance and security, particularly given that the authors' own test collection was predominantly nighttime (200 of 300 pairs).
• Consumer mobile thermal accessories, since the reference hardware is a sub-100-dollar thermal module paired with an ordinary smartphone.

Industry relevance. The method requires no factory calibration between the thermal and RGB sensors, tolerates unsynchronized captures, and runs as a one-step diffusion model — all properties that matter for shipping a feature on a phone rather than a lab prototype. The demonstrated downstream gains in zero-shot detection and segmentation also suggest value for teams building perception stacks rather than just image-enhancement products.

Future Directions

• Extending beyond two cameras. The formulation explicitly notes that the image-number dimension M can be extended to multiple RGB images, which raises the question of how CSM scales with several reference views.
• Video and temporal capture. The current work handles single RGB–thermal pairs with static misalignment augmentation; real handheld video would stress temporal consistency and continuous parallax in ways the current augmentation does not simulate.
• Closing the remaining gap on fidelity metrics. 3M-TI leads on perceptual metrics, but PSNR and SSIM are still highest for CoRPLE in the reported table, leaving open how to gain perceptual realism without sacrificing pixel-level fidelity.
• Dependence on external components and data. Text prompts come from RAM and downstream evaluation uses Grounded-SAM, so robustness depends on those models; the paper also notes that limited thermal data scale and diversity motivated the augmentation strategy in the first place.

Target Audience

Researchers and engineers working on thermal infrared imaging, RGB-guided super-resolution, cross-modal fusion, and diffusion-based image restoration. It is also relevant to practitioners building mobile or embedded multimodal perception systems, and to readers interested in how calibration requirements can be replaced by learned attention-based alignment in multi-camera setups. A working understanding of diffusion models and transformer attention is needed to follow the method sections closely.

Authors’ abstract

The miniaturization of thermal sensors for mobile platforms inherently limits their spatial resolution and textural fidelity, leading to blurry and less informative images. Existing thermal super-resolution (SR) methods can be grouped into single-image and RGB-guided approaches: the former struggles to recover fine structures from limited information, while the latter relies on accurate and laborious cross-camera calibration, which hinders practical deployment and robustness. Here, we propose 3M-TI, a calibration-free Multi-camera cross-Modality diffusion framework for Mobile Thermal Imaging. At its core, 3M-TI integrates a cross-modal self-attention module (CSM) into the diffusion UNet, replacing the original self-attention layers to adaptively align thermal and RGB features throughout the denoising process, without requiring explicit camera calibration. This design enables the diffusion network to leverage its generative prior to enhance spatial resolution, structural fidelity, and texture detail in the super-resolved thermal images. Extensive evaluations on real-world mobile thermal cameras and public benchmarks validate our superior performance, achieving state-of-the-art results in both visual quality and quantitative metrics. More importantly, the thermal images enhanced by 3M-TI lead to substantial gains in critical downstream tasks like object detection and segmentation, underscoring its practical value for robust mobile thermal perception systems. More materials: https://github.com/work-submit/3MTI.