Action-Aligned Retrieval with Pairwise Multimodal Reranking for Text-Based Person Anomaly Search

AI summary

Overview

• Research area: Computer vision and cross-modal retrieval, specifically text-based person search extended to person anomaly search, combining vision-language representation learning, query reformulation, and multimodal large language model (MLLM) reranking.
• Technical level: Advanced. The paper assumes familiarity with contrastive vision-language pretraining, SigLIP2-style sigmoid losses, Kullback-Leibler divergence consistency terms, bipartite ranking metrics (Recall@K, mAP), and MLLM prompting.
• Scope: The paper proposes ActPair, a three-stage coarse-to-fine framework that fine-tunes a vision-language model with action-tag supervision, fuses retrieval from an original query and an LLM-generated context-grounded rewrite, and applies training-free pairwise MLLM reranking via a pivot-promote algorithm, evaluated on the PAB benchmark and transferred to RSTPReid.

What This Paper Is About

Text-based person search normally retrieves people by appearance, but in surveillance the person of interest often looks like many others and can only be distinguished by a fine-grained, context-dependent behavior, such as "a man falling off a bicycle" versus "a man standing next to a fallen bicycle." Existing approaches rely on skeletal pose geometry that is ambiguous in a single static image, replace the original query with a rewrite and lose its raw detail, or score each candidate image independently with an MLLM. ActPair addresses all three problems in one pipeline by learning action-discriminative embeddings, retrieving with two complementary query views, and finally comparing candidate images directly against one another.

Key Contributions

1. Action-aligned multi-task representation learning for a VLM. The authors combine the native SigLIP2 pairwise sigmoid loss with explicit action-tag supervision and a cross-modal action-consistency term, so that representations encode context-conditioned action semantics rather than depending on pose geometry or background cues.
2. Context-grounded query rewriting with parallel late-fusion retrieval. An LLM injects canonical action and scene tags into a rewritten query, and the rewrite is fused at the score level with the original query instead of replacing it, retaining unadulterated fine-grained details while adding contextual grounding.
3. Efficient off-the-shelf pairwise multimodal reranking with a pivot-promote strategy. Using an MLLM as a direct pairwise comparator, the system reranks only the top-10 candidates rather than building a complete pairwise ordering. Under a matched Qwen3.5-9B comparison, direct candidate comparison improves R@1 and mAP over pointwise scoring.

Main Findings

• Best results among compared published baselines on the original PAB test. ActPair reaches 88.62% R@1, 99.75% R@5, 99.85% R@10, and 93.97% mAP.
• Improvements over the pose-aware CMP baseline. ActPair improves R@1 by 3.69 percentage points and mAP by 2.31 percentage points relative to CMP.
• Improvements over the cascade-based SSDC baseline. ActPair surpasses SSDC by 1.41 points in R@1 and 1.10 points in mAP.
• Gains concentrate at rank 1. R@5 and R@10 are described as already close to saturation, suggesting ActPair mainly improves fine-grained ordering of highly similar candidates rather than coarse candidate recall.
• Training-free methods lag far behind. Without domain adaptation, the strongest appearance-oriented method among IRRA, RaSa, and APTM reaches only 44.41% mAP (IRRA), while X-VLM reaches 83.96% mAP.
• Cross-dataset transfer to RSTPReid. Transferring from PAB to RSTPReid with the full pipeline yields 55.25% R@1, 69.05% R@5, and 73.75% R@10, versus 29.15% R@1, 50.40% R@5, and 60.75% R@10 for CMP under identical zero-shot transfer conditions. The paper notes that the lower R@5 and R@10 relative to CFine, APTM, and IRRA reflect the retrieval stage's coverage rather than a reranking limitation, since reranking only reorders the top 10.
• Action supervision beats pose supervision. Under a controlled SigLIP2 backbone, pose-based supervision (RTMPose) achieves 67.95% R@1 while action-based supervision achieves 72.14% R@1, a statistically significant paired-bootstrap gain of +4.19 (95% CI [+2.38, +6.07]); R@5 improves by +1.57 (CI [+0.51, +2.63]) and R@10 by +0.76 (CI [-0.10, +1.62]).
• Pairwise reranking beats pointwise reranking. With the same Qwen3.5-9B model and candidate pool, pointwise reranking reaches 80.13% R@1 and 87.70% mAP, while pairwise reaches 84.88% R@1 and 90.14% mAP, margins of +4.75 and +2.44 respectively (95% CI for R@1 [+3.08, +6.42]; for mAP [+1.45, +3.43]).
• Pairwise reranking also uses fewer model calls. Pairwise reranking averages 9.09 model calls per query versus 10.00 for pointwise, while exhaustive all-pairs comparison would incur quadratic cost.
• Progressive gains from each stage. On the challenge set, Recall@1 rises from 69.41 (CMP) to 72.14 (action-aligned retriever) to 75.73 (+ dual-view query) to 84.88 (+ pairwise reranking); on the public test the same progression is 84.93, 84.98, 85.95, and 88.62.
• Qualitative behavior. For queries 1 to 3, dual-view fusion improves candidate discovery and pairwise reranking promotes the ground-truth image to rank 1; Query 3 moves from rank 57 to 6 to 1 across the three stages. Query 4 is a failure case where the correct image is initially ranked first but demoted by later stages, indicating difficulty with long compositional descriptions involving action phases, multiple people, and precise spatial relations.

Methodology in Plain English

The system works in three stages.

Stage one: a retriever that understands actions. The authors start from a pretrained image-text model, SigLIP2-Patch16-256, and fine-tune it on the PAB training set. Alongside the model's usual image-text matching loss, they add a small classifier that predicts which action category an image or a caption belongs to, and a consistency term (a symmetric KL divergence) that pushes the image and text predictions to agree. This forces the model to use action evidence rather than just background scenery. Before training, the raw action and scene tags in the training annotations are cleaned up: only the most frequent tags are kept, and an LLM (Qwen3.5-9B) maps them into a fixed set of canonical categories, creating a standardized vocabulary.

Stage two: two query views instead of one. For each query, the system uses the canonical vocabulary to attach an action tag and a scene tag, then asks an LLM to rewrite the query into a fixed template: "The image shows [person description] wearing [clothing] [action]. The background features [scene details]." Crucially, the original caption is not thrown away. Both queries are run over the entire image gallery, the top-K = 100 candidates from each view are pooled into a union, and each candidate gets the arithmetic mean of its scores from the two branches. The list is re-sorted and the top M = 10 candidates are passed on.

Stage three: comparing candidates side by side. An off-the-shelf MLLM, Qwen3.5-9B, is prompted to compare two images at a time along fixed criteria - primary action, target person, clothing, object interaction, secondary people, scene and background, and spatial relationships - and returns which image is better or a tie. The pivot-promote algorithm takes the current top candidate as a pivot, compares it against all others, recursively sorts the subset judged better than the pivot, and concatenates the sorted better subset, the pivot, and the worse subset. Because it recurses only on candidates preferred to the pivot, expected comparison complexity is linear under a uniformly distributed pivot-rank assumption, though worst-case complexity remains quadratic; in practice, prior ordering keeps the recursive subsets small.

Implementation details: λ_action = 0.2; all Qwen inference uses deterministic decoding with temperature = 0 and seed = 0. Evaluation uses Recall@K (R@1, R@5, R@10) and mAP. Datasets are PAB (over 1 million image-text pairs spanning 1,600 anomalous and 1,000 normal behaviors, using an augmented version from AI City Challenge Track 4) and RSTPReid (20,505 images of 4,101 persons across 15 cameras).

Why This Matters

Impact on research. The paper frames the core obstacle as the "pose-semantic gap": the same skeletal configuration can correspond to different actions, so geometry alone cannot define behavior in a static image. Its controlled experiment showing action-tag supervision outperforming pose supervision on an identical backbone gives concrete evidence for that argument, and the pivot-promote scheme offers a cheaper alternative to exhaustive pairwise MLLM comparison. The work also argues against the common practice of replacing the original query during reformulation, showing value in keeping both views.

Real-world applications.

• Surveillance review, where an operator searches camera footage by describing a behavior rather than a face.
• Safety and incident triage, locating events such as falls or collisions among visually similar pedestrians.
• Retail or campus security, finding a person by what they were doing in a specific setting.
• General context-grounded retrieval beyond pedestrians, which the authors suggest could include distinguishing similar actions in sports.

Industry relevance. The method is built on off-the-shelf components (SigLIP2, Qwen3.5) and a training-free reranking stage, and it reduces MLLM calls per query from 10.00 to 9.09 in the top-10 setting, which matters for inference cost in deployed surveillance systems. The authors report infrastructure support from Saigon AI Hub, jointly established by VNG Group JSC and Vietnam National University Ho Chi Minh City, indicating industrial interest in the application area.

Future Directions

1. Removing the dependency on dataset-specific annotations. The authors state that constructing the canonical action and scene vocabularies currently requires dataset-specific annotations, which limits scalability in unannotated domains, and propose an unsupervised MLLM pipeline to parse raw captions into action and scene clusters.
2. Handling long compositional descriptions. The failure case in the qualitative analysis shows the model can demote a correct top-ranked image when queries involve action phases, multiple people, and precise spatial relations, which the paper identifies as an unresolved weakness.
3. Improving retrieval coverage in cross-domain transfer. On RSTPReid the lower R@5 and R@10 relative to CFine, APTM, and IRRA are attributed to the retrieval stage's coverage rather than reranking, leaving room to strengthen the first stage.
4. Rethinking the computational worst case. Pivot-promote has expected linear comparison complexity under a uniformly distributed pivot-rank assumption, but its worst-case complexity remains quadratic; the paper does not report latency or wall-clock inference measurements, so the practical cost profile of the reranking stage is not reported.

Target Audience

Researchers and practitioners working on text-based person search, cross-modal retrieval, and surveillance video analytics will benefit most, particularly those interested in vision-language model fine-tuning for fine-grained behavior discrimination or in MLLM-based reranking strategies. It is also relevant to engineers building real-world person or event retrieval systems who need to weigh reranking accuracy against multimodal inference cost. Readers without a background in contrastive retrieval and MLLM prompting will find the method sections demanding, since the paper assumes familiarity with sigmoid contrastive losses, KL-divergence consistency terms, and standard Recall@K and mAP evaluation.

Authors’ abstract

Text-based person anomaly search requires distinguishing individuals based on fine-grained, context-dependent behaviors rather than mere appearance. Existing methods struggle to capture these context-conditioned actions, frequently relying on isolated skeletal geometry, discarding raw query details during reformulation, or utilizing absolute pointwise scoring for multimodal verification. To address these limitations, we propose \textbf{ActPair}, a unified three-stage coarse-to-fine framework that combines action-aligned retrieval with pairwise multimodal reranking to bridge the pose-semantic gap. First, we fine-tune a vision-language model (VLM) with an action-aligned multi-task objective that encourages the representations to encode action-discriminative semantics. Second, we perform parallel late-fusion retrieval using the original query and a large language model (LLM)-generated context-grounded rewrite, retaining complementary details from both semantic views. Finally, we propose an efficient off-the-shelf reranking module that leverages a pivot-promote algorithm to perform direct pairwise visual comparisons, mitigating residual spatial and compositional ambiguities without the prohibitive inference costs of exhaustive evaluation. Extensive experiments demonstrate that our framework achieves the best results among the compared methods on the Pedestrian Anomaly Behavior (PAB) public test and transfers effectively to an unseen, non-anomaly-specific dataset.