AnchorHOI: Zero-shot Generation of 4D Human-Object Interaction via Anchor-based Prior Distillation

AI summary

Overview

Research area: Computer vision, specifically text-driven 4D (animated 3D) human-object interaction (HOI) generation using diffusion-model priors.

Technical level: Advanced. The paper assumes familiarity with score distillation sampling (SDS), Neural Radiance Fields (NeRFs), parametric human body models (SMPL-X), and image/video diffusion models.

Scope: The paper proposes AnchorHOI, a zero-shot framework that generates a static 3D human-object interaction and then animates it into a 4D sequence, using intermediate "anchors" (an anchor NeRF and anchor keypoints) to transfer priors from pre-trained image and video diffusion models.

What This Paper Is About

Generating realistic 4D human-object interaction (for example, a person sitting on a chair or playing a guitar) from a text description normally requires large paired text-HOI datasets collected with motion capture, which are scarce and expensive. Prior zero-shot work distill priors only from image diffusion models and, in doing so, leaves the human locked in a canonical pose and derives motion from a text-to-human motion model with no object awareness. AnchorHOI's goal is to generate interaction-aware 4D HOI from natural language alone, without any paired text-HOI or mocap data, by also exploiting video diffusion models and by routing those priors through intermediate anchors that make the high-dimensional optimization tractable.

Key Contributions

1. Anchor-based prior distillation for zero-shot 4D HOI. AnchorHOI advances zero-shot text-driven 4D HOI generation by exploiting hybrid priors from both pre-trained image diffusion models (IDMs) and video diffusion models (VDMs), using intermediate anchors instead of direct high-dimensional optimization.

2. Anchor NeRF for static interaction composition. A human-isolate NeRF is extracted from a coarsely optimized entangled human-object NeRF and serves as a bridge between image-diffusion priors and the articulated SMPL-X model. Skeleton alignment between the SMPL-X avatar and the anchor NeRF lets the framework optimize human pose and articulation rather than only relative position, producing expressive static 3D HOI.

3. Anchor keypoints for motion synthesis. Body keypoints (2D keypoints detected by OpenPose, reordered to match SMPL-X joint definitions) and contact keypoints (derived from geometric proximity and normal-alignment constraints on the object and human meshes) are combined into anchor keypoints that recover motion information lost to inter-subject occlusion in generated videos, enabling interaction-faithful 4D synthesis.

4. Extensive evaluation. Qualitative and quantitative evaluations show AnchorHOI substantially outperforms existing methods in both static 3D and dynamic 4D HOI generation, with superior diversity and generalization.

Main Findings

• 4D CLIP score: AnchorHOI achieves the highest mean CLIP score among the compared 4D methods, with a semantic score of 0.3149 versus 0.3017 for TC4D and 0.2833 for DreamGaussian4D (Table 1).

• 4D GPT-4V selection: AnchorHOI is selected 70.83% of the time on semantics and 54.17% on contact accuracy, compared with 16.67% and 20.83% for TC4D and 12.50% and 25.00% for DreamGaussian4D (Table 1).

• 4D user studies: AnchorHOI scores highest on every reported perceptual dimension: semantic 4.794, contact 4.756, penetration 4.673, motion 4.874, and overall 4.833, versus TC4D (3.119, 2.321, 2.863, 2.398, 3.664) and DreamGaussian4D (2.339, 2.382, 1.594, 2.634, 3.339) (Table 1).

• Static 3D CLIP score: AnchorHOI (static) reaches 0.3173, higher than MVDream (0.2948), InterFusion (0.2951), and AvatarGO (static) (0.2615) (Table 2).

• Static 3D GPT-4V selection: AnchorHOI (static) is preferred 57.89% on semantics and 52.63% overall, versus 21.05% and 26.32% for InterFusion, 15.79% and 10.53% for MVDream, and 5.26% and 10.53% for AvatarGO (static) (Table 2).

• Anchor NeRF ablation: Without the anchor NeRF, GPT-4V selects the result only 5.88% of the time, versus 94.12% for the full static setting; the paper reports that human pose fails to converge and position rather than articulation is optimized, while the anchor NeRF enables stable convergence and complex poses such as a crossed-legs posture (Table 3, Figure 6a).

• Anchor keypoint ablation: Removal of body keypoints drops GPT-4V selection to 5.89% and removal of contact keypoints to 17.65%, versus 76.47% for the full setting; without body keypoints postures are implausible, and without contact keypoints results may look plausible but lack meaningful physical contact (Table 3, Figure 6b).

• Baseline limitations observed: TC4D treats the scene holistically and lacks component-wise motion; DreamGaussian4D struggles to animate composited HOIs even with the same generated videos because it relies only on RGB and mask cues; AvatarGO fixes the human in a canonical pose and, since its 4D component is not publicly released, only its static 3D results were compared.

Methodology in Plain English

AnchorHOI splits the problem into two stages: first build a static 3D human-object interaction, then animate it.

For the static stage, the method optimizes a single entangled human-object NeRF under the guidance of image diffusion models, then extracts the human part as an "anchor NeRF" using a multi-view feature alignment loss. A canonical human avatar in a standing pose is generated first so that shape and appearance stay structurally complete and identity-consistent with the text. The remaining pose and placement parameters are then fitted by projecting canonical views of the anchor NeRF, detecting 2D skeletons with OpenPose, and minimizing the discrepancy between projected 3D SMPL-X joints and those detected 2D keypoints across multiple views using a robust Geman-McClure error. The object is initialized from the segmented object portion of the anchor NeRF, completed with SDS, and given an interaction-preserving SDS term, then extracted as a mesh with marching cubes.

For the animated stage, a video diffusion model generates a video from the prompt and a rendered static HOI image. Because subjects occlude each other in compositional scenes, the method does not rely on pixel-level video cues. Instead it extracts two kinds of anchor keypoints: body keypoints from OpenPose (18 per frame with confidence scores, reordered to match SMPL-X joints) and contact keypoints identified on the meshes by requiring both close geometric distance and antiparallel normals between human and object surfaces. Motion-dependent parameters (global rotations and translations of human and object, plus articulation pose) are optimized to track these anchors, while motion-invariant quantities (human scale, human shape and appearance parameters, and object identity) are inherited from the static stage and held fixed across frames. The total objective combines a keypoint term, a contact term, a penetration penalty, and regularization terms that include rendered-frame-to-video-frame error, a self-penetration penalty, and temporal smoothness.

Implementation details reported: DeepFloyd and a multi-view-consistent image diffusion model for SDS gradients, a video diffusion model to generate 5-second sequences, SMPL-X and VPoser as human body priors, OpenPose for body joints, Grounded-SAM for instance segmentation masks, Adam with a learning rate of 0.01, 3,000 iterations for interaction composition and 1,000 for motion synthesis, on an NVIDIA A6000 GPU. The paper does not report a dataset size for evaluation.

Why This Matters

Impact on research. The work shows a route to 4D HOI generation that avoids paired text-HOI and mocap data entirely, and argues that image-diffusion priors alone are insufficient for interaction: richer priors from video diffusion models are needed, and the anchors (NeRF and keypoints) are what make those priors usable. It also points out concrete shortcomings in prior zero-shot work, namely fixed canonical human poses and motion sources with no object awareness, and shows that contact and articulation can be recovered through geometric and skeleton cues rather than pixels.

Real-world applications (as motivated by the paper):

• AR/VR content creation
• Video games
• Embodied AI
• Robotics

Industry relevance. The framework produces 4D virtual-world content directly from natural-language prompts on a single NVIDIA A6000 GPU, which is relevant to studios and developers who need scalable animation of humans interacting with objects without motion-capture pipelines. Because it is zero-shot, it can be applied to object and interaction categories never seen in paired HOI training data.

Future Directions

• Handling intermittent contact. The current method assumes continuous contact between humans and objects, which overlooks the dynamic nature of contact in real-world scenarios.
• Articulated objects. The method is currently designed for rigid objects; extending it to objects with kinematic properties would be a meaningful step toward practical applications.
• Extending the anchor strategy. Since anchors are the mechanism that makes prior distillation tractable, a natural question is whether additional or different anchor types could cover interaction patterns the current anchor NeRF and anchor keypoints do not represent.
• Diversity and generalization. The paper claims superior diversity and generalization, but the evaluation metrics reported (CLIP score, GPT-4V selection, and user studies) do not quantify diversity or generalization directly, leaving open how these properties should be measured.

Target Audience

Researchers and graduate students in computer vision and graphics working on 3D/4D content generation, diffusion-model-based score distillation, human motion and avatar modeling, and compositional scene synthesis. It is also relevant to practitioners in AR/VR, gaming, and embodied AI who need to generate human-object interactions without motion-capture data, provided they already understand SDS, NeRFs, and SMPL-X.

Authors’ abstract

Despite significant progress in text-driven 4D human-object interaction (HOI) generation with supervised methods, the scalability remains limited by the scarcity of large-scale 4D HOI datasets. To overcome this, recent approaches attempt zero-shot 4D HOI generation with pre-trained image diffusion models. However, interaction cues are minimally distilled during the generation process, restricting their applicability across diverse scenarios. In this paper, we propose AnchorHOI, a novel framework that thoroughly exploits hybrid priors by incorporating video diffusion models beyond image diffusion models, advancing 4D HOI generation. Nevertheless, directly optimizing high-dimensional 4D HOI with such priors remains challenging, particularly for human pose and compositional motion. To address this challenge, AnchorHOI introduces an anchor-based prior distillation strategy, which constructs interaction-aware anchors and then leverages them to guide generation in a tractable two-step process. Specifically, two tailored anchors are designed for 4D HOI generation: anchor Neural Radiance Fields (NeRFs) for expressive interaction composition, and anchor keypoints for realistic motion synthesis. Extensive experiments demonstrate that AnchorHOI outperforms previous methods with superior diversity and generalization.