DuGI-MAE: Improving Infrared Mask Autoencoders via Dual-Domain Guidance

AI summary

Overview

Research area: Computer vision, specifically self-supervised learning for infrared (thermal) imagery.

Technical level: Intermediate. Readers should be familiar with transformer-based vision models and the masked autoencoder paradigm, though the paper's core ideas can be understood without deep mathematics.

Scope: This paper introduces an infrared foundation model (DuGI-MAE) pretrained on a new 590,700-image infrared dataset, using entropy-based token masking and a frequency-domain guidance module to overcome the specific challenges of infrared imagery.

What This Paper Is About

Infrared cameras capture heat rather than reflected light, which makes them valuable in darkness, fog, and rain, but their images contain far less texture and detail than ordinary photographs. Existing self-supervised models such as Masked Autoencoders (MAE), which learn by hiding parts of an image and reconstructing them, were designed for visible-light photos and therefore perform poorly on infrared data — the earlier infrared-specific model, InfMAE, still misses important regions when masking, ignores global relationships between distant image parts, and fails to account for the uneven sensor noise that plagues thermal cameras. The authors set out to build a better infrared foundation model by redesigning both how tokens are selected for masking and how the decoder is guided during reconstruction.

Key Contributions

1. Entropy-based deterministic masking. Instead of randomly masking patches (as in MAE) or sampling them at a fixed stride (as in InfMAE), the authors score every image patch by its Shannon entropy and keep the highest-entropy quarter. This guarantees that informative regions — thermal targets and target–background boundaries — are never accidentally discarded.

2. A Dual-Domain Guidance (DDG) module. Placed between the encoder and decoder, this module combines spatial-domain features with frequency-domain features processed by an Adaptive Frequency-Domain Modulation (AFDM) filter. The learnable radial filter suppresses low-frequency non-uniform noise (such as temperature drift) while preserving discriminative mid- and high-frequency thermal signals.

3. The Inf-590K dataset. A large-scale infrared pretraining corpus of 590,700 images gathered from multiple public sources plus self-collected footage, spanning 445 unique resolutions, diverse platforms (UAV, ground, vehicle, ship), varied weather, and day/night conditions. Redundant images were removed via feature-similarity filtering.

4. Broad empirical validation. DuGI-MAE is tested on three downstream tasks — infrared object detection, semantic segmentation, and infrared small-target detection — and consistently outperforms both fully supervised and self-supervised baselines.

Main Findings

• Object detection on M3FD-inf: With a Mask R-CNN head, DuGI-MAE reaches 59.1 mAP and 89.7 AP₅₀, surpassing InfMAE (57.1/87.9) by 2.0 and 1.8 points. It also leads with a Cascade R-CNN head (57.3/86.9 vs. InfMAE's 56.5/86.3).

• Semantic segmentation on MSRS: Using an FCN head, DuGI-MAE achieves 73.1 mIoU / 80.8 mAcc, and with UperNet it reaches 75.0 mIoU / 83.6 mAcc — both the best among all compared methods.

• Small-target detection on IRSTD-1K: DuGI-MAE records 67.1 mIoU, edging out InfMAE (66.5) and MCMAE (64.3), demonstrating that the dual-domain design helps when targets occupy only a few pixels.

• Larger pretraining data helps everyone: All self-supervised methods (MAE, MCMAE, InfMAE) improved when switched from the smaller Inf30 dataset to Inf-590K, confirming the value of scale for infrared pretraining.

• Entropy masking beats alternatives: An ablation shows entropy-based masking (59.1 mAP) outperforms random masking (57.2) and gray-value-based information-aware masking (58.4).

• DDG is model-agnostic: Plugging the DDG module into MAE, MCMAE, and InfMAE improved all three (e.g., MAE rose from 52.2 to 53.1 mAP; MCMAE from 56.8 to 57.2), indicating it is a general-purpose enhancement rather than something tied to one architecture.

Methodology in Plain English

The approach follows the masked autoencoder recipe: hide most of an image, then train a model to reconstruct the hidden parts. Three design choices distinguish DuGI-MAE.

First, the masking is entropy-driven rather than random. The image is split into patches, each patch's pixel-intensity histogram is used to compute a Shannon entropy score, and the top 25% highest-entropy patches are kept visible while the rest are masked. High-entropy patches tend to contain thermal targets and edges; low-entropy patches tend to be flat background.

Second, a frequency-domain filter cleans up the signal. Infrared images suffer from smooth, large-scale distortions (non-uniform noise) that dominate the low-frequency content of the Fourier spectrum. The authors apply a learnable radial filter — essentially a soft, tunable high-pass filter — that dampens the spectral center while preserving peripheral frequencies. This filtered version is converted back to the image domain and used as an additional "guide" for the transformer's attention, so spatial features pay more attention to target regions.

Third, the model uses a hierarchical three-stage encoder (2, 2, and 11 transformer layers) so that multi-scale features (F₁ through F₄) are available for downstream detectors and segmentation heads.

Pretraining used 400 epochs on four RTX 4090 GPUs with a 75% mask ratio, AdamW optimizer, and mean squared error loss. Downstream evaluation fine-tuned the pretrained encoder with standard heads (Mask R-CNN, Cascade R-CNN, FCN, UperNet, IRSTD).

Why This Matters

Impact on research: This work demonstrates that modality-specific foundation models are necessary for infrared vision — visible-light pretraining cannot simply be transferred. It also shows that frequency-domain techniques, long used in classical signal processing, can be productively embedded inside a self-supervised reconstruction pretext task.

Real-world applications:

• Nighttime and adverse-weather surveillance: Thermal cameras paired with DuGI-MAE backbones can detect intruders, vehicles, or vessels when visible-light systems fail.

• Autonomous driving in fog or darkness: Infrared pedestrian and vehicle detection is critical for safety, and improved small-target detection directly benefits distant obstacle recognition.

• Maritime and aerial monitoring: Ship detection and UAV-based reconnaissance rely on spotting small thermal signatures against cluttered backgrounds, exactly where DuGI-MAE shows gains.

• Search and rescue: Detecting people in smoke, darkness, or wilderness settings benefits from more discriminative thermal representations.

Industry relevance: The Inf-590K dataset and released code lower the barrier for companies building infrared perception systems, and the DDG module's plug-and-play nature means existing MAE-style pipelines can be upgraded without retraining from scratch.

Future Directions

• Incorporating physical priors: The authors explicitly state that future work will embed more physical knowledge about thermal radiation and sensor physics into the pretraining objective.

• Extending beyond static images: Infrared video, where temporal coherence and motion could supplement spatial cues, remains unexplored and is a natural next step.

• Scaling further: Whether returns continue to grow with datasets larger than 590K images, or whether architectural changes become the bottleneck, is an open question.

• Cross-modal pretraining: Joint visible–infrared pretraining could exploit the complementary strengths of both modalities, particularly for tasks like multispectral detection.

• Generalizing the entropy criterion: Whether Shannon entropy is optimal, or whether learned importance scores would work better, remains untested.

Target Audience

This paper is best suited for computer vision researchers and graduate students working on self-supervised learning, vision transformers, or infrared/thermal imaging. Practitioners building thermal detection or segmentation systems — particularly in surveillance, autonomous driving, or remote sensing — will find the pretrained model and Inf-590K dataset directly useful. Readers unfamiliar with masked autoencoders or Fourier analysis will need some background to follow the technical sections, though the core intuition is accessible without deep prerequisites.

Authors’ abstract

Infrared imaging plays a critical role in low-light and adverse weather conditions. However, due to the distinct characteristics of infrared images, existing foundation models such as Masked Autoencoder (MAE) trained on visible data perform suboptimal in infrared image interpretation tasks. To bridge this gap, an infrared foundation model known as InfMAE was developed and pre-trained on large-scale infrared datasets. Despite its effectiveness, InfMAE still faces several limitations, including the omission of informative tokens, insufficient modeling of global associations, and neglect of non-uniform noise. In this paper, we propose a Dual-domain Guided Infrared foundation model based on MAE (DuGI-MAE). First, we design a deterministic masking strategy based on token entropy, preserving only high-entropy tokens for reconstruction to enhance informativeness. Next, we introduce a Dual-Domain Guidance (DDG) module, which simultaneously captures global token relationships and adaptively filters non-uniform background noise commonly present in infrared imagery. To facilitate large-scale pretraining, we construct Inf-590K, a comprehensive infrared image dataset encompassing diverse scenes, various target types, and multiple spatial resolutions. Pretrained on Inf-590K, DuGI-MAE demonstrates strong generalization capabilities across various downstream tasks, including infrared object detection, semantic segmentation, and small target detection. Experimental results validate the superiority of the proposed method over both supervised and self-supervised comparison methods. Our code is available in the supplementary material.