VLM-as-a-Judge for Person Re-Identification

AI summary

Overview

• Research area: Computer vision — cloth-changing person re-identification (CC-ReID) and vision-language models (VLMs) used as inference-time arbiters.
• Technical level: Intermediate. Readers should know what retrieval metrics (rank-1, mAP) mean and be roughly familiar with Re-ID pipelines and VLM prompting; the paper is otherwise self-contained.
• Scope (one sentence): The paper shows that a frozen, general-purpose VLM, used zero-shot to rerank a short merged candidate list from two frozen CC-ReID experts, outperforms both experts and every naive ensemble of the two on all four scenarios of the CHIRLA benchmark, with a disagreement gate that removes 40% of the judge calls.

What This Paper Is About

Cloth-changing re-identification asks a system to match the same person across cameras when their clothing may have changed completely, and no single expert model wins consistently: SapiensID reaches 72.6% mAP on PRCC but 17.8% on LTCC, while DIFFER reaches 31.6% on LTCC but 64.7% on PRCC. The authors argue that this reversal means the two experts rely on different identity cues and therefore make complementary mistakes. Their goal is to exploit that complementarity at inference time — without training anything on the target data — by having a general-purpose VLM judge which of the two experts' candidate crops is the right person.

Key Contributions

1. A zero-shot VLM judge as an arbitration ensemble. A frozen VLM reranks the union of two frozen experts' top-k candidates and outperforms both experts and every naive ensemble of the two by a wide and consistent margin on all four CHIRLA scenarios, with no component trained or fine-tuned on the evaluated data.

Authors’ abstract

Cloth-changing person re-identification (CC-ReID) is still very challenging, and no method clearly dominates: methods that lead on one benchmark collapse on the others, because they rest on different identity cues. We show that a general-purpose vision-language model (VLM), used zero-shot as a judge for reranking a short candidate list, beats the expert retrieval models it reranks by a wide and consistent margin. On the four scenarios of CHIRLA, a long-term CC-ReID benchmark that no publicly released Re-ID model is trained on, reranking the candidates of two frozen experts raises rank-1 on every scenario, by 12.9 points on average over the best expert and 9.4 over the best naive ensemble of the two, with no component fine-tuned on the target data. Merging both experts' shortlists matches or beats reranking the stronger expert's shortlist alone, at lower cost. Introducing a VLM does, however, come at a cost: almost all of the pipeline's FLOPs are spent on it. We therefore borrow a criterion from ensembling and invoke the VLM judge only where the two experts disagree at rank 1. This threshold-free gate removes 40% of the calls, and a comparable share of total compute, with no significant drop in rank-1, trading 1.8 to 3.8 points of mAP. Judging remains the dominant cost, but on a benchmark where dedicated experts stall, we find a frozen general-purpose model to be the strongest judge of identity. The code is available at https://github.com/bdager/JudgeReID.