AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation

AI summary

Overview

Research area: Computer vision and computer graphics — specifically monocular (single-camera) 3D reconstruction of hand-object interaction (HOI), with applications to robotics, VR, and physics simulation.

Technical level: Advanced. The paper assumes familiarity with neural rendering, Structure-from-Motion, diffusion models, Vision-Language Models, MANO hand models, and differentiable optimization losses.

Scope: The paper presents AGILE, a framework that replaces reconstruction-based pipelines with an "agentic generation" pipeline to produce watertight, textured, simulation-ready hand and object assets from a single monocular video.

What This Paper Is About

Reconstructing how a hand manipulates an object from an ordinary monocular video is hard because the hand frequently occludes the object and the object may be textureless or moving quickly. Existing methods fail in two ways: neural-rendering methods produce fragmented, non-watertight geometry under occlusion, and nearly all of them depend on Structure-from-Motion (SfM, e.g., COLMAP) for initialization, which collapses on in-the-wild footage. AGILE instead uses a Vision-Language Model (VLM) to supervise generative models that synthesize a complete 3D object, then tracks that object across the video without any SfM.

Key Contributions

1. First agentic HOI pipeline. A VLM acts as supervisor, selecting informative keyframes to guide multi-view synthesis and acting as a critic that filters generated views and refined textures via rejection sampling. This produces high-fidelity, watertight object meshes independent of video occlusion.

2. SfM-free anchor-and-track optimization. Object pose is initialized only once, at a single "interaction onset frame" (IOF), using a foundation model (FoundationPose), then propagated temporally through semantic and geometric alignment — eliminating the brittle SfM dependency entirely.

3. Contact-aware interaction optimization. A composite objective combining mask alignment, DINOv3 semantic feature consistency, and an interaction stability loss that penalizes interpenetration and locks the object to the grasping hand parts.

4. Broad validation including real-to-sim transfer. Experiments span single-hand benchmarks (DexYCB, HO3D-v3), bimanual interactions (ARCTIC), and in-the-wild videos, plus real-to-sim retargeting onto a dexterous robotic hand.

Main Findings

• State-of-the-art geometry on DexYCB. AGILE achieves a Chamfer Distance (CD) of 0.52 cm² versus MagicHOI's 2.05 cm² — a reduction of nearly 75% — with F@5mm of 83.21% and F@10mm of 95.43%.

• State-of-the-art on HO3D-v3. AGILE reaches MPJPE 3.92 mm, CD 0.27 cm², F@5mm 86.63%, F@10mm 97.77%, and CD_h 15.81 cm².

• Dramatically better interaction stability. On DexYCB, MagicHOI exhibits a Hand-relative Chamfer Distance (CD_h) of 661.90 cm², indicating the object drifts far from the hand; AGILE achieves 94.60 cm², confirming that contact-aware optimization keeps the object anchored.

• 100% success rate versus high baseline failure. On DexYCB, MagicHOI fails on 75% of sequences and HOLD fails on 55%; AGILE succeeds on 100%. Baseline metrics are reported only over their successful subsets (the paper calls this "survivor bias"), yet AGILE's MPJPE of 19.06 mm still beats MagicHOI's 21.20 mm even though AGILE is averaged over harder samples.

• Robustness on the harder benchmark. On HO3D-v3, HOLD reports a 100% success rate and MagicHOI 83.3%, while AGILE also achieves 100%.

• Bimanual generalization. On the ARCTIC rigid-object subset, AGILE outperforms both HOLD and BIGS: MPJPE_l 25.0, MPJPE_r 23.8, CD_o 1.12, F@5 57.6, CD_l 21.9, CD_r 30.6, versus HOLD (27.1, 24.7, 2.07, 37.1, 105.9, 123.5) and BIGS (34.1, 36.1, 1.36, 56.4, 46.1, 31.3).

• Ablations confirm gains come from the agentic design, not just the base models. Removing agentic multi-view generation degrades HO3D CD from 0.27 to 2.85 and F@5mm from 86.63 to 30.12. Replacing the VLM critic with a max-mask heuristic gives CD 2.04 and F@5mm 76.51; keeping VLM selection but dropping multi-view synthesis gives CD 2.61 and F@5mm 66.58.

• Texture refinement matters for tracking, not just looks. Removing texture refinement raises CD_h from 15.81 to 20.18, because both FoundationPose initialization and the DINO-based loss depend on discriminative surface features.

• Removing the interaction loss causes the largest interaction degradation. Without L_interact, CD_h jumps from 15.81 to 54.40 on HO3D. Removing L_joint raises MPJPE from 3.92 to 5.06; removing L_mask raises CD from 0.27 to 0.40 and CD_h to 23.99; removing L_dino raises CD to 0.34 and CD_h to 21.64.

• Real-to-sim validation. The paper reports real-to-sim retargeting results showing stable kinematic transfer of reconstructed human hand–object interactions to a multi-fingered robotic hand without physics-based correction.

Methodology in Plain English

Step 1 — Build a good object mesh. Because the hand blocks the object, the pipeline asks a VLM to pick the most informative keyframes from the video (typically 1–4). Those frames prompt an image generator to invent orthogonal views of the object. A second VLM role — the critic — scores each generated view against the original video on geometry, texture, and material correspondence, and anything below a strict consistency threshold is thrown away and regenerated. The surviving views are lifted into 3D by a feed-forward generator, then cleaned up with automated retopology and UV unwrapping. Finally, an image-to-image editing model sharpens the texture using the high-resolution multi-views, again supervised by the VLM critic.

Step 2 — Set up metric scale and a single pose anchor. The system assumes a fixed camera (camera coordinates are the world frame). It uses MoGe-2 for camera intrinsics and a metric depth map, and SAM2 for hand and object masks. The hand is fit with WiLoR to get MANO parameters, then its scale is recovered by aligning the MANO mesh to depth-unprojected point clouds with constrained ICP, and its translation by PnP against 2D keypoints. The object's global scale is recovered the same way across all frames, and its pose is initialized once at the interaction onset frame using FoundationPose on the pre-scaled mesh.

Step 3 — Track outward from the anchor. Starting at the interaction onset frame, optimization propagates both forward to the end of the video and backward to the beginning. Each frame is solved in two steps: first, the hand translation is refined using a joint reprojection loss against 2D keypoints; second, with the hand fixed, the object's rigid pose is optimized using three losses — silhouette alignment, DINOv3 semantic feature similarity (to survive occlusion and textureless regions), and an interaction stability term that expresses hand vertices in the object's local frame and penalizes their displacement between consecutive frames, weighted by how close each hand vertex is to the object surface (via the object's signed distance function).

Why This Matters

Impact on research. The paper argues for a paradigm shift from reconstruction to agentic generation for interaction learning. It shows that a VLM-supervised generative loop can substitute for photogrammetry-style multi-view optimization, and that dropping SfM initialization removes the single most common point of catastrophic failure in HOI pipelines. It also reframes texture quality as a tracking prerequisite rather than a cosmetic detail.

Real-world applications:

• Robot imitation learning — producing simulation-ready manipulation trajectories from internet videos of human hands.
• Digital twins and VR/AR content — generating clean, watertight, textured interaction assets from ordinary footage.
• Physics-based simulation and animation — feeding physics engines that require explicit, non-interpenetrating topology, which implicit neural representations do not provide.
• Real-to-sim-to-real transfer — the paper demonstrates retargeting reconstructed interactions onto a multi-fingered robotic hand without physics-based correction.

Industry relevance. Scaling robot manipulation data is a known bottleneck; a method that turns arbitrary monocular video into simulation-valid assets without a fragile SfM stage directly addresses that bottleneck. The reliance on off-the-shelf foundation models (WiLoR, SAM2, MoGe-2, FoundationPose, DINOv3) also means the approach is composable with future upgrades to those components.

Future Directions

• Extending beyond rigid objects. The ARCTIC evaluation uses only the rigid-object subset, and the method models object pose as a rigid transform with an anisotropic scale; articulated or deformable objects are not addressed.
• Scaling and cost of the agentic loop. The VLM supervision and rejection sampling add generative inference and iteration; the main paper states that runtime analysis is provided in the supplementary material but does not report it in the content analyzed here.
• Reducing dependence on the single interaction onset frame anchor. Errors at the IOF propagate bidirectionally; a mechanism for detecting or correcting a bad anchor is not reported.
• Generalizing the keyframe selection policy. The current design selects 1–4 keyframes; whether the VLM critic and keyframe budget generalize to longer, more cluttered, or multi-object sequences is not established.

Target Audience

Researchers and graduate students in computer vision, computer graphics, and robotics working on 3D hand-object interaction, 4D reconstruction, generative 3D asset creation, or robot learning from demonstration. It is also relevant to graphics engineers building simulation-ready assets and to practitioners combining VLMs with generative 3D models for embodied AI, though the density of geometric losses and foundation-model components makes it a demanding read for newcomers.

Authors’ abstract

Reconstructing dynamic hand-object interactions from monocular videos is critical for dexterous manipulation data collection and creating realistic digital twins for robotics and VR. However, current methods face two prohibitive barriers: (1) reliance on neural rendering often yields fragmented, non-simulation-ready geometries under heavy occlusion, and (2) dependence on brittle Structure-from-Motion (SfM) initialization leads to frequent failures on in-the-wild footage. To overcome these limitations, we introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning. First, we employ an agentic pipeline where a Vision-Language Model (VLM) guides a generative model to synthesize a complete, watertight object mesh with high-fidelity texture, independent of video occlusions. Second, bypassing fragile SfM entirely, we propose a robust anchor-and-track strategy. We initialize the object pose at a single interaction onset frame using a foundation model and propagate it temporally by leveraging the strong visual similarity between our generated asset and video observations. Finally, a contact-aware optimization integrates semantic, geometric, and interaction stability constraints to enforce physical plausibility. Extensive experiments on HO3D, DexYCB, ARCTIC, and in-the-wild videos reveal that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior arts frequently collapse. By prioritizing physical validity, our method produces simulation-ready assets validated via real-to-sim retargeting for robotic applications. Project page: https://agile-hoi.github.io.