UniHOI: Unified Human-Object Interaction Understanding via Unified Token Space

AI summary

Overview

Research area: Computer Vision — specifically human-object interaction (HOI) understanding, sitting at the intersection of multimodal large language models (MLLMs), HOI detection, and text-to-image generation.

Technical level: Advanced. The paper assumes familiarity with MLLM architectures, discrete visual tokenization (VQGAN), cross-attention mechanisms, and semi-supervised/cycle-consistency training paradigms.

Scope: The paper proposes UniHOI, a single semi-supervised MLLM that jointly performs HOI detection and HOI image generation by mapping both visual tokens and interaction semantics into one shared discrete token vocabulary.

What This Paper Is About

HOI detection (identifying ⟨human, action, object⟩ triplets in images) and HOI generation (synthesizing images from a specified interaction) are inverse tasks that share underlying semantics, yet existing work treats them as separate problems. Detection models only recognize and cannot generate, while generation models typically depend on explicit spatial constraints such as bounding boxes, limiting generalization. UniHOI asks whether both tasks can be unified in one framework so that they share knowledge, reduce dependence on fine-grained annotations, and generalize to open-world, long-tailed interactions.

Key Contributions

1. A modality-aware unified token space that places visual codebook tokens and semantic (text/HOI triplet) tokens in a single discrete vocabulary with a shared embedding matrix, enabling direct bidirectional mapping and compositional reasoning between detection and generation — going beyond embedding-level alignment used in prior multimodal work.

2. An Interaction-Aware Attention (IAA) module built on a parameter-shared, symmetric cross-attention block. The direction of attention is determined only by the task (visual tokens as queries for detection, semantic tokens as queries for generation), making the same module serves both inverse tasks.

3. A unified semi-supervised learning strategy based on dual cycle consistency in the shared token space, which allows joint training from strongly-supervised, weakly-supervised, and unlabeled data without handcrafted pseudo-labels.

4. State-of-the-art empirical results on both sides of the task, with reported gains of 4.9% accuracy on long-tailed HOI detection and a 42.0% boost in interaction metrics on open-vocabulary generation.

Main Findings

• HOI detection results: UniHOI reaches 48.16 mAP on the HICO-DET full test set, 50.74 mAP on rare categories, and 51.34 mAP under the Known Object setting. It outperforms the strongest listed prior method, HOI-IDiff (47.71 mAP full), on these metrics.

• V-COCO results: UniHOI achieves 72.91 on AP^S1_role and 77.45 on AP^S2_role. The paper notes that its AP^S1_role is slightly lower than HOI-IDiff's 73.42, while AP^S2_role is the highest reported.

• HOI generation results: UniHOI reports the best Image Reward (1.17), lowest FID (18.2), and highest CLIP Score (32.46) among compared methods, plus the top HOI Score (0.64) and Interaction Accuracy (0.54). The paper states this surpasses the previous state of the art by 14.3% on HOI Score and 42.0% on Interaction Accuracy.

• GenAIEval submetrics: UniHOI reports 0.99 on Single Obj., 0.76 on Two Obj., and 0.42 on Position, which the authors describe as new state-of-the-art across all GenAIEval submetrics.

• Unified token space matters: Ablating the token space shows Full mAP dropping from 48.16 (full model) to 47.62 ("w/o Type Emb.") and 47.03 ("Separate Emb."), while HOI Score drops from 0.64 to 0.57 and 0.44 respectively, and Interaction Accuracy from 0.54 to 0.46 and 0.32.

• IAA matters: Removing IAA lowers Full mAP to 47.10, HOI Score to 0.51, and Interaction Accuracy to 0.39; removing modality-type embedding from the symmetric module yields 47.71 mAP, 0.59 HOI Score, and 0.49 Interaction Accuracy.

• More weak and unlabeled data helps both tasks: With 100% strong supervision only, HOI detection is 45.62 mAP and HOI generation HOI Score is 0.22. Moving to 50%/20%/30% (strong/weak/unsupervised) gives 46.43 and 0.38; 20%/20%/60% gives 47.65 and 0.59; and 10%/25%/65%, the final setting, gives 48.16 and 0.64. The authors interpret this as mutual promotion between the two tasks.

• Qualitative behavior: The paper reports that UniHOI handles hard detection cases with better fine-grained perception and generates more natural interactions, including realistic hand poses and tool usage such as picking up a paintbrush.

Methodology in Plain English

The authors start from the observation that a HOI triplet and the image depicting it are two sides of the same concept. Their idea is to convert both into the same kind of "word" — discrete tokens — so a single Transformer can read or write either one.

Concretely, images are converted into discrete visual tokens using a VQGAN tokenizer (the one from Chameleon), and text or structured HOI triplets are converted into semantic tokens. Both sets of tokens draw on one shared, trainable embedding matrix. Because visual and semantic tokens otherwise look alike, each token also carries a learnable modality-type embedding that says whether it came from the visual or the semantic side, and the final input embedding is the sum of the two. This lets the model tell the modalities apart and control generation, which the authors point out is something Liquid does not do when it merges vocabularies.

The IAA module is a single set of cross-attention parameters used in both directions. For detection, visual tokens act as queries and semantic tokens as keys and values; for generation, the roles swap. Attention is computed with the modality-type embeddings added to queries and keys, and a residual connection preserves the original query information. Because it is symmetric and parameter-shared, the same block supports both tasks and inputs can swap roles.

Training is semi-supervised and built on cycle consistency. If the model detects a triplet from an image and then generates an image back from that triplet, the reconstruction should match the original — and vice versa. A unified loss combines these cycle losses (with weight 1 for each pathway) with a semantic alignment term and a diversity term computed from attention maps. Because everything lives in the same token space, paired, weakly-labeled, and unlabeled data can all be trained on without handcrafted pseudo-labels.

The training mixture is 10% strongly-supervised data (HICO-DET and V-COCO, which provide triplets with boxes and role labels), 25% weakly-supervised data (150K image-text pairs from LAION-SG where scene-graph triplets act as weak signals), and 65% unsupervised data (400K image-text pairs from LAION-400M, captions only). The backbone is Llama3-8B with VQGAN from Chameleon as the image tokenizer. Training uses Adam with a constant learning rate of 5×10⁻⁴, a linear warm-up over the first 10,000 iterations, 700,000 total iterations, per-device batch size 8 on 32 NVIDIA H800 GPUs, gradient accumulation with step size 2 to reach an effective batch size of 16, and a temperature of τ = 0.07. A hybrid data loading strategy balances labeled and unlabeled samples within each mini-batch.

Why This Matters

Impact on research. The paper challenges the standard division of labor in HOI research, where detection and generation are studied as separate problems with separate annotations. It shows that a shared discrete token space can host both, and that supervision from one task appears to help the other. It also connects HOI research to the MLLM-based unified understanding-and-generation line of work (MMaDA, Liquid, Chameleon, Show-o, Janus), while adding explicit interaction structure that those general models lack.

Real-world applications.

• Assistive and robotics systems that must both recognize what a person is doing with an object and imagine or predict plausible interaction outcomes.
• Content creation and advertising, where a user specifies an interaction in text and needs a faithful image of that specific interaction, including correct hand-object contact.
• Dataset and annotation scaling: because the method trains on 10% strong / 25% weak / 65% unlabeled data, it offers a route to build HOI systems where exhaustive triplet annotation is impractical.
• Surveillance, sports analysis, and safety monitoring, where recognizing rare or long-tailed interactions (a category the paper specifically reports gains on) is often the hard case.

Industry relevance. The method is built on an 8B-parameter open backbone (Llama3-8B) and runs on 32 H800 GPUs for 700,000 iterations, which is within reach of well-resourced industrial labs but not of small teams. The semi-supervised recipe directly targets a real industry bottleneck — annotation cost — and the unified model replaces two separate systems with one.

Future Directions

1. Scaling the unified token space. The paper does not report how performance changes with tokenizer size, vocabulary size, or backbone scale; exploring this is a natural next step.
2. Pushing further into open-vocabulary and long-tail territory. The reported gains on rare HICO-DET categories and open-vocabulary generation suggest headroom remains, especially for interactions absent from HICO-DET, which the authors claim their mined data partially covers.
3. Efficiency of semi-supervised training. The authors mention a hybrid data loading strategy to mitigate inefficiency from semi-supervised learning, implying that training cost under mixed supervision remains an open concern.
4. Extending the formulation beyond static images. The unified formulation maps images, triplets, boxes, captions, and prompts; video or multi-step interaction sequences are not addressed and would extend the inverse-task symmetry considerably.

Target Audience

Researchers and graduate students working on human-object interaction, multimodal large language models, and unified understanding-and-generation architectures. It is also relevant to practitioners in image generation who need interaction-faithful outputs rather than layout-conditioned ones, and to applied teams looking for semi-supervised recipes that reduce fine-grained annotation requirements. Readers without background in MLLMs, discrete visual tokenization, or cross-attention would find the method sections demanding.

Authors’ abstract

In the field of human-object interaction (HOI), detection and generation are two dual tasks that have traditionally been addressed separately, hindering the development of comprehensive interaction understanding. To address this, we propose UniHOI, which jointly models HOI detection and generation via a unified token space, thereby effectively promoting knowledge sharing and enhancing generalization. Specifically, we introduce a symmetric interaction-aware attention module and a unified semi-supervised learning paradigm, enabling effective bidirectional mapping between images and interaction semantics even under limited annotations. Extensive experiments demonstrate that UniHOI achieves state-of-the-art performance in both HOI detection and generation. Specifically, UniHOI improves accuracy by 4.9% on long-tailed HOI detection and boosts interaction metrics by 42.0% on open-vocabulary generation tasks.