Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection

AI summary

Overview

Research area: Computer vision — specifically zero-shot Human-Object Interaction (HOI) detection using vision-language models and prompt learning.

Technical level: Advanced. The paper assumes familiarity with CLIP, prompt learning (e.g., CoOp, MaPLe), two-stage HOI detectors such as DETR, and evaluation protocols for zero-shot composition.

Scope: The paper introduces VDRP, a prompt-learning framework that combines visual diversity-aware prompts with region-aware prompt augmentation to improve zero-shot HOI detection on the HICO-DET benchmark.

What This Paper Is About

Zero-shot HOI detection requires a model to localize a person and an object in an image and name the interaction between them, even when that specific verb-object pair was never seen during training. Existing CLIP-based prompt-learning methods typically use one static text prompt per verb, which fails when the same verb appears in many different poses and contexts (intra-class visual diversity) and when different verbs look visually similar (inter-class visual entanglement). This paper's goal is to build prompts that carry both a sense of how much a verb's appearance varies and what the human, object, and union regions specifically look like.

Key Contributions

1. Visual diversity-aware prompt learning. The authors inject group-wise visual variance into the learnable context embedding and apply Gaussian perturbation scaled by that variance, so the prompts represent not just the average meaning of a verb but its expected visual spread.
2. Region-aware prompt augmentation. Region-specific concepts are retrieved from a pool generated for the human, object, and union regions and added to the diversity-aware prompts, producing three region-specific prompt sets that improve discrimination among visually similar verbs.
3. A unified framework (VDRP) integrating both modules, evaluated on the HICO-DET benchmark under four zero-shot settings and reported to achieve state-of-the-art results with only 4.50M trainable parameters.
4. A quantitative characterization of the two challenges, using a diversity score over frozen CLIP CLS features (verbs 0.364 ± 0.060 versus objects 0.274 ± 0.048) and t-SNE projections showing overlapping verb clusters.

Main Findings

• Verbs are more visually diverse than objects. Using a diversity score defined as the expectation of 1 − cos(·) over frozen CLIP CLS features, verb classes score 0.364 ± 0.060 while object classes score 0.274 ± 0.048, supporting the claim that a single static embedding per verb is insufficient.
• Verb features are entangled across classes. t-SNE projections of union-region CLIP features for five randomly selected verbs show substantial overlap not only at the sample level but also across verb prototypes, indicating poor inter-class separability.
• Best results under NF-UC and RF-UC. On HICO-DET with a Resnet50+ViT-B backbone, VDRP reaches NF-UC HM 33.85 (Full 32.57, Unseen 36.45, Seen 31.60) and RF-UC HM 32.77 (Full 33.78, Unseen 31.29, Seen 34.41). In NF-UC this is +5.01 Unseen and +4.31 HM over CLIP4HOI; in RF-UC it is +2.27 Unseen and +1.59 HM over EZ-HOI.
• Best results under UO. With 4.50M trainable parameters, VDRP reaches HM 34.41 (Full 33.39, Unseen 36.13, Seen 32.84), outperforming CMMP and EZ-HOI by +1.97 and +2.27 HM respectively.
• Best results under UV. VDRP reaches HM 29.80 (Full 32.73, Unseen 26.69, Seen 33.72), a +1.59 gain on unseen verbs over EZ-HOI.
• Parameter efficiency. The method uses 4.50M trainable parameters, compared with 6.85M for EZ-HOI, 56.7M for CLIP4HOI, 2.30M for CMMP, 42.05M for GEN-VLKT, and 66.18M for HOICLIP.
• Both modules are complementary. In the ablation, going from the static prompt baseline to the full VDRP model improves every setting — for example, NF-UC Full goes from 28.99 (BASE) to 30.17 (+VDP), 30.43 (+RAP), and 32.57 (+VDRP); RF-UC Unseen from 25.64 to 29.16 (+VDP), 26.46 (+RAP), and 31.29 (+VDRP).
• Group size of 5 works best. Comparing a single global variance against groups of 3 and groups of 5 verbs, Group_5 yields the best overall numbers (for example NF-UC HM 32.57, Unseen 36.45), while group-wise variance generally improves over the global estimate.
• Injection scale α = 0.02 is best. Tested values {0.01, 0.02, 0.10}; the authors state 0.02 matches the initialization scale of CLIP's context embeddings.
• Gaussian perturbation helps. Comparing β ∈ {0, 0.1}, adding perturbation improves most unseen scores (for example NF-UC HM 33.85 versus 33.57 without; UO HM 34.41 versus 33.67 without).
• Sparsemax is the most consistent retrieval strategy. Tested against Softmax and Top-3, Sparsemax suppresses irrelevant concepts and emphasizes informative ones.
• Augmentation scale γ = 0.2 is best. Tested values {0.2, 0.5, 1.0}; γ = 1.0 slightly degrades performance across multiple settings, which the authors attribute to noisy region concepts.
• Region concepts come from LLMs. Concepts are generated with LLaMA-7B and GPT-4; the authors note LLaMA-7B produced noise and redundancy in object-related concepts, so GPT-4 was additionally used, with details in the supplementary material.

Methodology in Plain English

The pipeline is two-stage. First, a frozen DETR detector finds people and objects in an image and produces a confidence score, a class embedding, and a bounding box for each. Second, a frozen CLIP image encoder with lightweight adapters extracts features for three regions — the human box, the object box, and their union box — via RoIAlign. A "spatial head" refines the union feature using the human and object features plus their boxes (details are in the supplementary material).

On the text side, the method builds a prompt for each verb in two steps:

1. Add diversity. For each verb, the authors crop union boxes from the training set, extract CLIP CLS tokens, and compute a mean and variance. Because rare or unseen verbs have too few samples, each verb borrows variance statistics from a group of semantically similar verbs (chosen by cosine similarity between CLIP text embeddings), giving a group-wise variance. A small MLP turns that variance into a modulation vector that is added to the shared learnable context embedding, scaled by a small factor α. After the CLIP text encoder produces the prompt embedding, Gaussian noise scaled by the group-wise standard deviation (and scaled by β) is added, so the prompt represents a spread of appearances rather than a single point.
2. Add regions. Using LLMs, the authors generate K short visual concepts per verb for each of the three region types, and encode them with the CLIP text encoder to form concept pools. For a given region feature, cosine similarity to each concept is computed, Sparsemax converts those scores into sparse weights (assigning exact zeros to uninformative concepts), and the weighted sum of concepts is added to the diversity-aware prompt with a scaling factor γ. This yields three prompt sets — human, object, union.

Verb classification logits are the dot products of each region feature with its region prompt; the final HOI logit is the average of the three, and the model is trained with focal loss for multi-label verb classification.

Why This Matters

The paper argues that progress in zero-shot HOI detection has been limited by an over-simplified view of verbs — one prompt per verb — while verbs are the most visually variable part of the interaction vocabulary. By modeling variance explicitly and grounding prompts in region-level concepts, VDRP suggests a path to better generalization without large numbers of trainable parameters.

Real-world applications:

• Assistive and monitoring systems that must recognize what a person is doing with an object (for example, in elder care or safety monitoring), including interactions not present in the training data.
• Robotics and human-robot collaboration, where a robot needs to interpret the interaction between a person and a tool or object.
• Content understanding and retrieval for image or video archives, where fine-grained action labels are needed for rare verbs.
• Dataset curation and annotation assistance, since zero-shot models can propose labels for long-tail interaction categories.

Industry relevance: The method builds on frozen, widely available components (DETR, CLIP ViT-B/16) and reports only 4.50M trainable parameters, which keeps training inexpensive relative to methods such as CLIP4HOI (56.7M) or HOICLIP (66.18M). The reported performance is measured only on HICO-DET, so transfer to other domains is not established in the paper.

Future Directions

• Relax the distributional assumptions. The authors acknowledge in the paper checklist that limitations such as the Gaussian assumption in visual space are discussed in the supplemental material; testing non-Gaussian or learned variance models is a natural next step.
• Reduce reliance on LLM-generated concepts. Since the quality of region concepts depends on the LLM used (LLaMA-7B produced noise and redundancy for object concepts, prompting the use of GPT-4), alternatives such as retrieval from image-caption corpora or learned concept pools could be explored.
• Validate beyond HICO-DET. All reported experiments use HICO-DET only; evaluating on other HOI benchmarks or in open-world settings would test whether the gains transfer.
• Understand the region decomposition more deeply. The paper reports the three region prompts (human, object, union) and their averaged logits; how much each region contributes per verb, and whether region weighting should be adaptive, remains unexplored.

Target Audience

Researchers and graduate students working on HOI detection, vision-language models, and prompt learning, particularly those interested in zero-shot compositional recognition. It is also useful for practitioners who want a parameter-efficient way to adapt CLIP-style models to fine-grained interaction classification, and for readers who want a compact quantitative argument for why verbs — not objects — are the hard part of HOI detection.

Authors’ abstract

Zero-shot Human-Object Interaction detection aims to localize humans and objects in an image and recognize their interaction, even when specific verb-object pairs are unseen during training. Recent works have shown promising results using prompt learning with pretrained vision-language models such as CLIP, which align natural language prompts with visual features in a shared embedding space. However, existing approaches still fail to handle the visual complexity of interaction, including (1) intra-class visual diversity, where instances of the same verb appear in diverse poses and contexts, and (2) inter-class visual entanglement, where distinct verbs yield visually similar patterns. To address these challenges, we propose VDRP, a framework for Visual Diversity and Region-aware Prompt learning. First, we introduce a visual diversity-aware prompt learning strategy that injects group-wise visual variance into the context embedding. We further apply Gaussian perturbation to encourage the prompts to capture diverse visual variations of a verb. Second, we retrieve region-specific concepts from the human, object, and union regions. These are used to augment the diversity-aware prompt embeddings, yielding region-aware prompts that enhance verb-level discrimination. Experiments on the HICO-DET benchmark demonstrate that our method achieves state-of-the-art performance under four zero-shot evaluation settings, effectively addressing both intra-class diversity and inter-class visual entanglement. Code is available at https://github.com/mlvlab/VDRP.