Hierarchical Prompt Learning for Image- and Text-Based Person Re-Identification

AI summary

Overview

• Research area: Computer Vision — person re-identification (ReID), specifically unifying image-to-image (I2I) and text-to-image (T2I) retrieval in a single framework using prompt learning with a CLIP backbone.
• Technical level: Advanced. The paper assumes familiarity with vision transformers, CLIP, contrastive losses, and prompt tuning.
• Scope: The paper proposes Hierarchical Prompt Learning (HPL), a unified framework combining a Task-Routed Transformer, hierarchical identity/instance prompts, and cross-modal prompt regularization, evaluated across six ReID benchmarks.

What This Paper Is About

Image-based ReID retrieves a person from a query image, while text-based ReID retrieves a person from a natural-language description. Both share identity cues such as clothing and gender, but text queries also carry instance-specific details (actions, carried objects) that image-based training ignores — and the authors show that simply training both tasks in one model degrades performance on both. The goal is a single architecture that handles both retrieval tasks without the tasks interfering with each other.

Key Contributions

1. A unified ReID framework that jointly handles image-to-image (I2I) and text-to-image (T2I) retrieval within one architecture, rather than treating the tasks separately.
2. A task-aware prompting mechanism combining a Task-Routed Transformer (TRT) with dual classification tokens in the shared CLIP visual encoder, plus a Hierarchical Prompt Learning (HPL) scheme that mixes identity-level learnable tokens with instance-level pseudo-text tokens.
3. Cross-Modal Prompt Regularization (CMPR), which aligns instance-level prompt tokens generated from visual and textual modalities in the textual prompt space.
4. Extensive benchmark evaluation across six datasets, reporting state-of-the-art performance on both I2I and T2I tasks.

Main Findings

• Joint training conflict is real: The authors report that naively training I2I and T2I in a single model produces a noticeable performance drop for both tasks versus training them separately, which they attribute to semantic conflict between identity-level cues (shared) and instance-specific attributes (T2I-only).
• T2I results: HPL achieves Rank-1 / mAP of 76.28 / 70.90 on CUHK-PEDES, 66.61 / 44.14 on ICFG-PEDES, and 64.00 / 53.13 on RSTPReID.
• I2I results: HPL achieves Rank-1 / mAP of 95.99 / 89.82 on Market1501, 91.04 / 79.01 on MSMT17, and 90.35 / 82.93 on DukeMTMC-ReID — surpassing CLIP-ReID, which reports 95.50 / 89.60, 88.70 / 73.40, and 90.00 / 82.50 respectively.
• TRT effectiveness: Adding the dual class token design gives a 1.07% improvement in T2I Rank-1 accuracy and a 2.07% increase in I2I mAP over the single-class-token baseline.
• HPL and CMPR effectiveness: On CUHK-PEDES + Market1501, adding CMPR on top of TRT and HPL yields +1.01% on T2I Rank-1 and +0.84% on I2I mAP. The full combination progresses from 74.22 T2I Rank-1 (baseline) to 75.27 (TRT) to 75.60 (TRT+HPL) to 76.28 (TRT+HPL+CMPR).
• Uni-modal vs. dual-modal prompts: Vision-guided and text-guided instance prompts each beat the baseline, but the dual-modal setting brings a slight drop in I2I Rank-1 (-0.12%) while improving mAP (+0.13%); the T2I impact is marginal.
• Attention behavior differs by task: Grad-CAM visualizations show I2I tokens focus on identity-related regions such as clothing and body shape, while T2I tokens emphasize text-mentioned details — for example, in the second row, T2I attention highlights a grey shoulder bag that I2I ignores.
• Hyperparameters: The loss weights λ1 = 0.4 and λ2 = 0.06 give the best results; both smaller and larger values degrade performance.

Methodology in Plain English

The researchers start from a CLIP model and modify its visual encoder by adding a second classification token. One token is dedicated to the T2I task and supervised with cross-modal similarity distribution matching and cross-modal identity classification losses; the other is dedicated to I2I and supervised with identity classification and triplet ranking losses. This keeps one shared backbone while letting each task pull features in its own direction.

For prompting, they use a template: "A photo of [id-tokens] and [inst-tokens] person." The identity tokens are fixed learnable parameters shared across samples. The instance tokens are generated on the fly: modality-specific inversion networks (four transformer layers, ViT-block style) convert either an image feature or a text feature into pseudo-text tokens that are inserted into the template. These pseudo-prompts are then re-encoded by a text encoder, and an inversion consistency loss checks that the reconstructed prompt still matches its source feature.

Training runs in two stages. Stage I ("prompt construction") trains only the inversion networks and prompts while the encoders stay frozen, using contrastive losses on identity-level prompts plus the inversion consistency loss. Stage II ("representation learning") trains the full pipeline with the base losses plus cross-modal identity classification, instance-level prompt alignment (both text-guided and vision-guided), and CMPR, which directly minimizes the squared Frobenius distance between visual-derived and text-derived instance prompts. Settings reported: 384×128 image inputs, 77 text tokens, 10 epochs in the construction stage with Adam, learning rates of 5×10⁻⁵ for the decoder and 0.02 for identity prompts with 0.8 exponential decay, and 60 epochs in the alignment stage with a 5-epoch linear warm-up from 10⁻⁶ to 10⁻⁵ followed by cosine annealing. Each batch mixes 64 T2I image-text pairs and 64 I2I images with 4 instances per identity. Test samples are removed from training splits and identity labels are unified across datasets. Training used a single NVIDIA RTX 4090 GPU.

Why This Matters

Person re-identification is a core building block for surveillance and public security systems, and real deployments rarely have only one query type available. This work shows that the two query modes can coexist in one model without sacrificing either, which matters for research on multi-task representation learning and for anyone building retrieval systems that must accept both photos and descriptions.

• Smart-city surveillance: Operators could search camera networks with either a photo of a person of interest or a written description such as clothing and carried objects.
• Retail and loss prevention: Staff could locate a person from a witness description when no image is available.
• Public safety and search operations: Missing-person searches could run on verbal descriptions that are converted into text queries rather than requiring a reference photograph.
• Robotics and assistive systems: A robot or assistant could be instructed in natural language to find a specific person in a scene.

Industry relevance: Because the method builds on CLIP and adds a second classification token rather than a second backbone, it is a relatively lightweight route to multi-task retrieval — the kind of consolidation that reduces serving cost when a company needs to support image and text search over the same gallery. The authors release code at https://github.com/LH-Z-Ac/HPL-AAAI26.

Future Directions

• Reducing the dual-modal trade-off: The dual-modal prompt setting slightly decreased I2I Rank-1 (-0.12%). Understanding and eliminating this small regression is an open problem.
• Extending beyond two query modalities: The framework handles image and text queries; whether the same task-routing and hierarchical prompt design extends to video, sketch, or audio descriptions is untested in this paper.
• Scaling the backbone and training data: The method uses CLIP pretrained on LUPerson and large-scale synthetic image-text pairs. Behavior under larger vision-language backbones is not reported.
• Cross-dataset identity unification: The authors unify identity labels across datasets and remove test samples from training splits; how robust this protocol is on larger, less curated real-world galleries is not evaluated here.

Target Audience

Researchers and graduate students working on person re-identification, cross-modal retrieval, or prompt-based adaptation of vision-language models. It is also relevant to practitioners who need one retrieval system to accept both image and text queries, and to readers interested in how task interference arises when two related vision tasks share a single backbone. Prior familiarity with CLIP, ViT architecture, and contrastive losses is assumed; the paper does not report the identity of benchmark dataset splits beyond sizes, and does not report inference latency or model size, which should be noted by readers looking for deployment cost figures.

Authors’ abstract

Person re-identification (ReID) aims to retrieve target pedestrian images given either visual queries (image-to-image, I2I) or textual descriptions (text-to-image, T2I). Although both tasks share a common retrieval objective, they pose distinct challenges: I2I emphasizes discriminative identity learning, while T2I requires accurate cross-modal semantic alignment. Existing methods often treat these tasks separately, which may lead to representation entanglement and suboptimal performance. To address this, we propose a unified framework named Hierarchical Prompt Learning (HPL), which leverages task-aware prompt modeling to jointly optimize both tasks. Specifically, we first introduce a Task-Routed Transformer, which incorporates dual classification tokens into a shared visual encoder to route features for I2I and T2I branches respectively. On top of this, we develop a hierarchical prompt generation scheme that integrates identity-level learnable tokens with instance-level pseudo-text tokens. These pseudo-tokens are derived from image or text features via modality-specific inversion networks, injecting fine-grained, instance-specific semantics into the prompts. Furthermore, we propose a Cross-Modal Prompt Regularization strategy to enforce semantic alignment in the prompt token space, ensuring that pseudo-prompts preserve source-modality characteristics while enhancing cross-modal transferability. Extensive experiments on multiple ReID benchmarks validate the effectiveness of our method, achieving state-of-the-art performance on both I2I and T2I tasks.