MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification

AI summary

Overview

Research area: Computer vision — multi-modal object re-identification (ReID), spanning multi-spectral person ReID and vehicle ReID.

Technical level: Advanced. The paper assumes familiarity with Vision Transformers, token-based architectures, contrastive/triplet metric learning, and standard ReID evaluation metrics (mAP, CMC).

Scope: The paper proposes MDReID, a single ViT-based "any-to-any" retrieval framework that supports arbitrary combinations of query and gallery modalities (RGB, NIR, TIR) by decomposing each modality into shared and specific feature components, validated on three multi-spectral benchmarks.

What This Paper Is About

Most multi-modal ReID methods assume that query and gallery images come from the same set of sensors — an assumption that rarely holds in deployment, where cameras differ in which spectral channels they provide. The paper's goal is a single model that works both when modalities match and when they do not (for example, an RGB query against a NIR gallery). The authors argue that forcing the model to predict a missing modality is ill-posed, and instead propose to split each modality's representation into a transferable shared part and a modality-unique specific part, then match accordingly.

Key Contributions

1. MDReID framework: A flexible any-to-any image-level object ReID framework that supports retrieval across arbitrary query-gallery modality combinations, rather than only modality-aligned settings.
2. Modality-Decoupled Learning (MDL): A strategy that inserts a modality-shared and a modality-specific learnable token into the ViT sequence for each modality, explicitly disentangling the two components, with a fixed-size zero-padded feature vector plus binary availability mask to handle any modality presence pattern.
3. Modality-aware Metric Learning (MML): Two complementary losses — a representation orthogonality loss (ROL) enforcing cross-modal consistency of shared features and orthogonality between shared and specific features, and a knowledge discrepancy loss (KDL) enforcing that the combined representation is more discriminative than either component alone.
4. Extensive evaluation: Experiments on three multi-spectral benchmarks (RGBNT201, RGBNT100, MSVR310) covering modality-matched, missing-modality, and modality-mismatched scenarios, reporting mAP improvements of 9.8%, 3.0%, and 11.5% in modality-matched settings and average gains of 3.4%, 11.8%, and 10.9% in modality-mismatched settings.

Main Findings

• Modality-matched gains: On RGBNT201 (person ReID), MDReID reports 82.1 mAP, 85.2 R-1, 90.3 R-5, and 92.6 R-10; the paper states this outperforms TOP-ReID's 72.3 mAP, 76.6 R-1, 84.7 R-5, and 89.4 R-10 by 9.8%, 8.6%, 5.6%, and 3.2% respectively.
• Vehicle ReID gains: MDReID reports the highest mAP on RGBNT100 (85.3% mAP, 95.6 R-1) and outperforms the next-best method on MSVR310 by 11.5% in mAP and 13.7% in R-1, reaching 51.0 mAP and 68.9 R-1.
• Missing-modality robustness: Under all six missing-modality configurations on RGBNT201, MDReID averages 54.4 mAP and 54.8 R-1, an average improvement of 10.0% mAP and 9.4% Rank-1 over TOP-ReID.
• Modality-mismatched robustness: Across RT-to-NT, RT-to-N, R-to-N, and R-to-NT, MDReID improves average mAP by 3.4% (RGBNT201), 11.8% (RGBNT100), and 10.9% (MSVR310) over the compared baselines, reaching average mAP/R-1 of 21.6/19.1, 38.6/47.4, and 22.1/31.7 on the three datasets.
• Single model beats specialists: Trained on RGBNT201, the single MDReID model outperforms four scenario-specific expert models by 9.2% mAP and 8.9% R-1 on average, while adapting to all four scenarios.
• Ablation — MDL: Adding modality decoupling to the base framework raises average mAP by 11.5% and Rank-1 by 11.1%.
• Ablation — ROL and KDL: ROL alone adds 1.8% mAP and 2.7% Rank-1; KDL alone adds 0.5% mAP and 1.7% Rank-1; combining both adds 3.8% mAP and 4.1% Rank-1. Going from index 3 (MDL+ROL) to index 5 (MDL+ROL+KDL) improves mAP by 2.0% and Rank-1 by 1.5%.
• Ablation — classifiers: Using modality-specific classifiers instead of a single classifier increases mean mAP by 13.4% and mean R-1 by 13.7%.
• Best hyper-parameters: With w1 = 1.5 and w2 = 5.25, ablations report the best average score of 43.2% mAP and 42.3% R-1 (with 50.2% Rank-5 and 54.9% Rank-10) across the evaluated scenarios.
• Full ablation ordering: Index 1 (no MDL, no ROL, no KDL) 27.8 mAP / 27.1 R-1; index 2 (MDL only) 39.4 / 38.2; index 3 (MDL+ROL) 41.2 / 40.8; index 4 (MDL+KDL) 39.9 / 40.9; index 5 (all) 43.2 / 42.3.
• Editor baseline limitation: EDITOR is described as designed exclusively for modality-matched settings and performs poorly under mismatch (for example 8.5 average mAP on RGBNT201 and 2.5 on MSVR310).
• Not reported in the provided content: The computational complexity analysis (trainable parameters and FLOPs) and additional visualizations are stated to be in the supplementary material, which is not included in the supplied text.

Methodology in Plain English

The method starts from a CLIP-Base vision transformer. Instead of the usual single classification token, the authors prepend two learnable tokens to every image: one intended to capture what is common across modalities, and one intended to capture what is unique to that modality. Each input modality (RGB, NIR, or TIR) therefore produces two feature vectors.

These vectors are assembled into a fixed-length list of six slots — three specific slots and three shared slots — in a fixed modality order. A binary availability mask of the same length records which slots are actually present; missing modalities are filled with zeros and their mask entries set to zero. Because the format is fixed, the same model can compare any query to any gallery, even when the two use different sensors.

The similarity between a query and a gallery is the average of two scores. The specific similarity only compares specific features of the same modality (RGB-to-RGB, and so on), divided by the number of jointly available specific slots. The shared similarity compares all available pairs of shared features across modalities via a small 3x3 matrix of dot products, masked so only jointly present pairs count.

During training, all modalities are assumed available, and two extra losses operate on the six-dimensional feature vector. The representation orthogonality loss compares the 6x6 matrix of pairwise cosine similarities against a target matrix that is an identity block among the three specific features, a zero block between specific and shared features, and an all-ones block among the three shared features — pushing specific features apart from each other and from shared features, while pulling shared features together. The knowledge discrepancy loss uses triplet-style distances to require that the combined shared-plus-specific representation has a smaller maximum positive distance and a larger minimum negative distance than either component alone. The two are combined as L_MML = w1 × L_ROL + w2 × L_KDL, and the total objective is L = L_ce + L_tri + L_MML, where L_ce is label-smoothing cross-entropy and L_tri is triplet loss.

Training used PyTorch on a single NVIDIA RTX 4090 with CUDA 12.5 and Python 3.8, Adam with batch size 64, a base learning rate of 3.5 × 10⁻⁴, a reduced 5 × 10⁻⁶ rate for the visual encoder, and 50 epochs. Inputs were resized to 256 × 128 for RGBNT201 and 128 × 256 for RGBNT100 and MSVR310, with random horizontal flipping, cropping, and erasing as augmentation.

Why This Matters

Impact on research: The paper challenges the modality-aligned assumption that underpins most multi-modal ReID work, and offers a decomposition-based alternative to the common "reconstruct the missing modality" strategy, which the authors argue is ill-posed. The masked six-slot representation is a reusable design pattern for other settings with heterogeneous or partially available sensors.

Real-world applications:

• Surveillance and security systems that mix RGB, near-infrared, and thermal cameras across a site, where a suspect seen by one camera type must be matched against footage from another.
• Vehicle re-identification for intelligent transportation, matching plates or vehicles captured by day cameras against night or thermal sensors.
• Search and rescue or perimeter monitoring at night, where NIR or TIR capture is the only option for one side of the retrieval.
• Edge deployments where only a subset of sensor types is available at each camera, and retraining a separate model per modality combination is impractical.

Industry relevance: The single-model, any-to-any property reduces the need to maintain a separate specialist model for each query-gallery sensor combination, which the authors directly contrast with the four expert models they trained. The use of an off-the-shelf CLIP-Base backbone and standard hardware (a single RTX 4090) makes the approach practical to reproduce.

Future Directions

1. Efficiency quantification: The paper defers computational complexity — trainable parameters and FLOPs — to supplementary material; a full accounting of the two extra tokens and the pairwise similarity computation would clarify deployment cost.
2. Beyond the three modalities: The framework is defined for RGB, NIR, and TIR, giving a six-slot vector and a 3x3 shared-similarity matrix. Extending to additional sensing modalities (for example depth or event cameras) raises questions about how the fixed structure and mask scale.
3. Modality combinations not evaluated: The mismatch experiments focus on four R-to-N-based configurations (RT-to-NT, RT-to-N, R-to-N, R-to-NT) and six missing-modality configurations on RGBNT201. Exhaustive coverage of all arbitrary combinations is not demonstrated.
4. Generalization to other ReID domains: Evaluation covers person (RGBNT201) and vehicle (RGBNT100, MSVR310) ReID; whether the decoupling transfers to other domains, or when the training-time assumption that all modalities are present is violated, remains an open question.

Target Audience

Researchers and engineers working on multi-modal person and vehicle re-identification, cross-spectral retrieval, and transformer-based representation learning. It is also relevant to practitioners deploying surveillance or transportation systems with heterogeneous camera fleets who cannot guarantee matched sensors between query and gallery, and to readers interested in feature disentanglement strategies for missing-modality robustness.

Authors’ abstract

Real-world object re-identification (ReID) systems often face modality inconsistencies, where query and gallery images come from different sensors (e.g., RGB, NIR, TIR). However, most existing methods assume modality-matched conditions, which limits their robustness and scalability in practical applications. To address this challenge, we propose MDReID, a flexible any-to-any image-level ReID framework designed to operate under both modality-matched and modality-mismatched scenarios. MDReID builds on the insight that modality information can be decomposed into two components: modality-shared features that are predictable and transferable, and modality-specific features that capture unique, modality-dependent characteristics. To effectively leverage this, MDReID introduces two key components: the Modality Decoupling Learning (MDL) and Modality-aware Metric Learning (MML). Specifically, MDL explicitly decomposes modality features into modality-shared and modality-specific representations, enabling effective retrieval in both modality-aligned and mismatched scenarios. MML, a tailored metric learning strategy, further enforces orthogonality and complementarity between the two components to enhance discriminative power across modalities. Extensive experiments conducted on three challenging multi-modality ReID benchmarks (RGBNT201, RGBNT100, MSVR310) consistently demonstrate the superiority of MDReID. Notably, MDReID achieves significant mAP improvements of 9.8\%, 3.0\%, and 11.5\% in general modality-matched scenarios, and average gains of 3.4\%, 11.8\%, and 10.9\% in modality-mismatched scenarios, respectively. The code is available at: \textcolor{magenta}{https://github.com/stone96123/MDReID}.