CLIP4VI-ReID: Learning Modality-shared Representations via CLIP Semantic Bridge for Visible-Infrared Person Re-identification

AI summary

Overview

Research area: Computer vision, specifically visible-infrared person re-identification (VI-ReID), combining cross-modal retrieval with vision-language pre-training (CLIP).

Technical level: Advanced. The paper assumes familiarity with CLIP, prompt learning, contrastive losses, transformer/ResNet backbones, and standard ReID evaluation metrics (CMC, mAP, mINP).

Scope: The paper proposes CLIP4VI-ReID, a three-stream, three-stage CLIP-driven network that uses text semantics generated only from visible images as a bridge to align visible and infrared pedestrian representations.

What This Paper Is About

Visible-infrared person re-identification requires matching images of the same person when the query and gallery come from different camera modalities (RGB versus thermal infrared). The paper argues that existing CLIP-based VI-ReID methods break down because CLIP was pre-trained on natural visible-light images, so generating text semantics directly from infrared images produces noisy, imprecise descriptions, and because those methods never adaptively adjust the text semantics they obtain. CLIP4VI-ReID addresses both problems by generating text semantics only for visible images and using them to correct infrared feature embeddings.

Key Contributions

1. A three-stream CLIP-based modality-shared representation network. CLIP4VI-ReID is built on a pre-trained CLIP image and text encoder, modified into a modality-specific RGB encoder, a modality-specific IR encoder, and a modality-shared encoder, with a three-stage learning strategy that performs coarse-to-fine RGB-IR cross-modal alignment.

2. Staged learning with stage-specific objectives and losses. Each stage has its own goal and loss function, and text semantics are generated only for visible images, avoiding the noise that comes from trying to generate infrared text semantics given the physical differences between natural and infrared images, while also simplifying model design.

3. Semantic information injected into visual representations. Text serves as a bridge for indirect visible-infrared alignment, and the high-level semantic alignment stage refines the text semantics so they contain only id-related information, improving discriminability of the learned modality-shared features.

4. Validation on two standard VI-ReID benchmarks. The authors report experiments on SYSU-MM01 and RegDB, comparing against state-of-the-art methods, and state that the code will be available at https://github.com/y0406/CLIP4VI-ReID.

Main Findings

• Infrared text semantics are unreliable: The paper argues that pre-trained CLIP cannot effectively align IR images and text because CLIP is trained on visible-light images rich in color, texture, and semantic information, whereas IR images mainly reflect temperature differences with different imaging principles and feature distributions. The authors state they provide representative failure cases in Fig. 9 where IR-derived text semantics fail to capture fine-grained ID-related attributes, while RGB-derived semantics remain more semantically consistent.

• Existing CLIP-based cross-modal methods leave two gaps: They assume CLIP can align IR images and text, which the authors say may not hold, and they do not adaptively adjust the generated text semantics, so inaccurate semantics propagate into later results. The paper also notes that additional fusion networks increase parameter count.

• Two-stage alignment before refinement is not enough: After the Text Semantic Generation and Infrared Feature Embedding stages, the paper reports that CLIP4VI-ReID still shows weak discriminative ability, attributed to text semantics containing excessive RGB modality-specific information, which is why the High-level Semantic Alignment stage is added.

• Reported benchmark performance: The paper states that CLIP4VI-ReID achieves superior performance than other state-of-the-art methods on widely used VI-ReID datasets, validated on two benchmarks. Comparison tables list results for methods including AlignGAN, XIV, DDAG, AGW, MPANet, SFANet, MID, FMCNet, SPOT, TSME, PMT, TOPLight, and CMTR. The provided paper excerpt is truncated inside the SYSU-MM01 table (at CMTR, TMM'2...), so the specific Rank-1, Rank-10, Rank-20, mAP, and mINP numbers for CLIP4VI-ReID are not reported in the available content, and neither are the RegDB results.

• Datasets used: SYSU-MM01 contains 15,792 infrared and 287,628 visible images of 491 pedestrian identities, captured by Camera 1 through Camera 6, with all-search and indoor-search modes, each in single-shot and multi-shot settings; the experiments use 96 identities as the testing set, including 3,803 infrared images for querying and 301 randomly selected visible images for gallery. RegDB contains 4,120 infrared and 4,120 visible images of 412 pedestrian identities, split into 206 training identities and 206 testing identities, evaluated in visible-to-infrared and infrared-to-visible settings, with all experiments repeated 10 times and the average recorded.

• Evaluation metrics: cumulative matching characteristic (CMC), mean Average Precision (mAP), and mINP.

Methodology in Plain English

The model splits CLIP's image encoder so that the first blocks extract modality-specific features and the last four blocks extract modality-shared features. This yields a modality-specific RGB encoder, a modality-specific IR encoder, and a shared encoder; the CLIP text encoder forms the text branch.

Stage 1 — Text Semantic Generation (TSG). The authors create a structured learnable description for each pedestrian identity: "a photo of a [X1], [X2], ..., [XM] person," where each bracket token is learnable and matches word-embedding dimensions. Text features and RGB image features are extracted, and a bidirectional contrastive loss (visible-to-text plus text-to-visible) matches them. During this stage the image and text encoders are frozen and only the text tokens are optimized, so the text encoder is used to its full semantic capacity. The resulting text features are saved for efficiency and reuse.

Stage 2 — Infrared Feature Embedding (IFE). Because the text semantics contain both high-level id-related semantics (learned by the shared encoder) and low-level modality-specific semantics (learned by the RGB encoder), this stage injects id-related semantics into the IR branch. The IR encoder and text encoder are frozen and only the shared encoder is trained, using a bidirectional IR-to-text and text-to-IR contrastive loss. This rectifies IR feature embeddings and gives preliminary visible-infrared alignment with text as the bridge.

Stage 3 — High-level Semantic Alignment (HSA). To fix weak discriminability, RGB-to-text and IR-to-text cross-entropy losses optimize the RGB encoder, IR encoder, text encoder, and text tokens while the already-tuned shared encoder stays fixed. An identity loss and a weighted regularization triplet (wrt) loss further sharpen visible and infrared features. The total objective combines both cross-entropy terms (weighted by balance hyper-parameters) with the identity and wrt losses.

Training and inference. Training proceeds in three stages over specified iteration counts. At inference, only the visual encoder features are used, with cosine similarity for identity recognition, so components other than the visual encoder are not needed.

Why This Matters

Impact on research: The paper challenges a common assumption in CLIP-based cross-modal ReID — that a frozen CLIP visual encoder can meaningfully describe infrared images in language. By restricting text generation to visible images and using text purely as a bridge, it offers an alternative design pattern for modality-shared representation learning, and it raises the question of how to adaptively refine generated semantics rather than accepting them as-is.

Real-world applications:

• Around-the-clock surveillance, where dual-mode cameras switch between RGB and thermal imaging depending on lighting.
• Tracking suspects or locating missing persons across camera networks in low-light or nighttime conditions.
• Public security and forensic search over mixed visible and thermal camera archives.
• Any retrieval system deployed in facilities that mix visible-light and infrared sensors.

Industry relevance: Most modern social surveillance systems already use dual-mode cameras that automatically switch based on ambient light, so a method that matches identities across those two modes directly targets an existing hardware reality. The design also matters operationally: inference requires only the visual encoder and cosine similarity, which simplifies deployment.

Future Directions

• Quantifying the infrared semantics problem: The paper's motivation leans on failure cases (Fig. 9) where IR-derived text semantics miss fine-grained id-related attributes; a systematic study of when and why this happens would clarify the limits of the approach.
• Adaptive semantic correction: The paper raises the risk of inaccurate text semantics but applies refinement in a fixed staged pipeline; learning to detect and correct bad semantics on the fly remains open.
• Scaling and generalization: Only SYSU-MM01 and RegDB are used in the reported experiments, so behavior on larger or more diverse cross-modal datasets, and under more complex scene conditions, is untested in the available content.
• Reducing reliance on staged training: Three sequential stages with separate iteration counts and balance hyper-parameters add tuning burden; whether a joint or fewer-stage formulation achieves comparable alignment is an open question.

Target Audience

Researchers and graduate students working on person re-identification, cross-modal retrieval, and vision-language models, particularly those extending CLIP to non-natural-image modalities. It is also relevant to practitioners building all-weather surveillance or identity-retrieval systems who need to understand how semantic supervision can bridge the visible-infrared gap.

Authors’ abstract

This paper proposes a novel CLIP-driven modality-shared representation learning network named CLIP4VI-ReID for VI-ReID task, which consists of Text Semantic Generation (TSG), Infrared Feature Embedding (IFE), and High-level Semantic Alignment (HSA). Specifically, considering the huge gap in the physical characteristics between natural images and infrared images, the TSG is designed to generate text semantics only for visible images, thereby enabling preliminary visible-text modality alignment. Then, the IFE is proposed to rectify the feature embeddings of infrared images using the generated text semantics. This process injects id-related semantics into the shared image encoder, enhancing its adaptability to the infrared modality. Besides, with text serving as a bridge, it enables indirect visible-infrared modality alignment. Finally, the HSA is established to refine the high-level semantic alignment. This process ensures that the fine-tuned text semantics only contain id-related information, thereby achieving more accurate cross-modal alignment and enhancing the discriminability of the learned modal-shared representations. Extensive experimental results demonstrate that the proposed CLIP4VI-ReID achieves superior performance than other state-of-the-art methods on some widely used VI-ReID datasets.