Reconstructing Humans and Objects in Interaction using Large Reconstruction Models

AI summary

Overview

Research area: 3D computer vision, specifically monocular 3D human-object interaction (HOI) reconstruction, combining parametric human body models with Large Reconstruction Models (LRMs).

Technical level: Intermediate. The paper is readable without deep math, but assumes familiarity with concepts such as SMPL-H, Chamfer distance, camera-frame fitting, and feed-forward 3D reconstruction backbones. The optimization details (latent pose priors, robust losses) are advanced.

Scope: The paper introduces MILO, a framework that turns a single LRM-generated human-object mesh into a structured 3D interaction reconstruction by segmenting the mesh and fitting a parametric body model (and optionally an object template) to it, evaluated on InterCap, HODome, and IMHD.

What This Paper Is About

Reconstructing how a person interacts with an object in 3D from a single photo is hard: depth is ambiguous, bodies and objects occlude each other, and objects come in unlimited shapes with no standard parametric model (unlike humans, who have SMPL-H). Existing methods typically fit parametric humans and retrieved object templates to 2D image cues, and many depend on privileged information such as ground-truth contact or depth annotations. MILO's goal is to sidestep much of this by using a Large Reconstruction Model to produce an image-specific combined human-object mesh, then treating that mesh as a geometric scaffold that already encodes relative arrangement and proximity between person and object. The reconstruction problem becomes one of interpreting the LRM mesh rather than fitting ambiguous 2D observations.

Key Contributions

1. Demonstrating LRMs as a scaffold for HOI reconstruction. Instead of optimizing against ambiguous reprojection objectives, the authors reframe the problem as explaining the LRM-generated human-object mesh, which preserves relative arrangement and proximity cues without needing contact annotations.

2. A robust fitting pipeline. MILO segments the LRM mesh into human and object components, estimates 3D keypoints via multi-view rendering and triangulation, and fits the SMPL-H body model in two stages (root fitting, then pose fitting), optionally aligning an object template via semantic correspondences.

3. State-of-the-art results with weaker supervision. MILO outperforms prior methods on InterCap, HODome, and IMHD while using no ground-truth contact information, and it works with or without an object template.

4. Validation of the design choices. Ablations show the joint human-object scaffold, not the fitting procedure alone, drives the accuracy gains, and that the approach is not tied to a specific LRM backbone.

Main Findings

• InterCap reconstruction (PA-CD, cm, lower is better): MILO with template scores 6.96 (human), 18.97 (object), 7.45 (combined); without template, 6.85, 20.74, 9.36. Prior methods: CONTHO 8.36/24.30/13.14, HOI-TG 8.22/25.05/14.63, PHOSA 10.07/23.36/13.38, Open3DHOI 6.88/31.18/10.17, PICO 7.43/21.85/10.33. MILO beats all prior work even in the template-free setting, and without ground-truth contact.

• HODome and IMHD: Against PICO on HODome, MILO with template scores 9.50/12.97/6.68 and without template 9.71/13.75/6.38, versus PICO's 13.12/14.44/10.07. On IMHD, MILO with template scores 11.76/13.39/10.10 and without template 8.99/9.71/6.98, versus PICO's 15.81/17.70/13.24.

• Contact estimation from geometry: Although MILO does not predict contact, proximity between the fitted SMPL-H mesh and the aligned object yields F1 30.00, Precision 36.57, Recall 39.49, and geometric error 39.57 cm on InterCap, compared to DECO's 9.37, 12.57, 10.28, and 126.70 cm.

• Backbone sensitivity: Swapping Hunyuan3D-2.0 for SAM3D (joint) gives comparable InterCap results (7.00/21.99/9.80 vs. 6.85/20.74/9.36), indicating MILO is not tied to a specific LRM. InstantMesh is worse (7.51/33.21/10.84), and an oracle ground-truth mesh reaches 3.28/4.93/3.12, showing remaining error is largely bounded by upstream reconstruction quality.

• Joint scaffold versus object-only pipelines: An EasyHOI-style object-only pipeline on InterCap scores 41.57/77.06/37.21 without a contact loss and 9.48/27.93/12.08 with one, while a SAM3D independent (separately reconstructed human and object composed via MoGe alignment) baseline scores 18.18/31.07/15.89. MILO's joint scaffold scores 6.85/20.74/9.36, confirming the scaffold rather than the fitting procedure is the main driver.

• Both fitting stages matter: On InterCap, no fitting at all yields 30.58/23.53/14.10 (template-free) and 14.97/37.02/14.57 (with template); root fitting improves these to 7.98/22.53/10.15 and 7.79/18.63/7.99; full pose fitting reaches 6.85/20.74/9.36 and 6.96/18.97/7.45.

• Qualitative generalization: MILO produces more coherent contact and fewer interpenetrations than PICO-fit on in-the-wild PICO-db images, HODome, IMHD, and InterCap.

Methodology in Plain English

The pipeline has five stages.

1. Produce a combined mesh. An RGBA image (where the alpha channel encodes the combined human and object segmentation) is passed to Hunyuan3D-2.0, which returns a single mesh containing both the person and the object they interact with. This mesh acts as a "scaffold" that already positions the two relative to each other.

2. Estimate 3D keypoints on that mesh. The mesh is rendered from 60 virtual viewpoints. A body pose estimator (ViTPose, giving 25 COCO body keypoints) and a hand estimator (HaMeR, giving 21 keypoints per hand) run on every rendered image. Low-confidence detections are discarded, surviving observations are triangulated across all pairs of views, and the hypothesis with the largest consensus set under a reprojection-error threshold is refined to yield 67 3D keypoints with aggregated confidence.

3. Fit the human body. Starting from HMR2.0 and HaMeR predictions, the authors first run a root fitting stage for 30 iterations that optimizes only global orientation and translation (plus a human scale parameter, since LRM meshes are not guaranteed to be at metric scale), using a Geman-McClure robust loss against the 3D keypoints. A second pose fitting stage runs for 60 iterations, additionally optimizing body shape, body pose and hand pose under VPoser and MANO priors, an L2 shape prior, an anchoring loss (tolerance margin τ = 2.5) that discourages large deviations from the HMR2.0 initialization, and a one-way robust Chamfer loss between observed LRM points and only the visible predicted vertices. Loss weights are λ_rf = 10, λ_bp = λ_hp = 0.04, λ_β = 0.05, λ_ha = 10, λ_3D = 50.

4. Segment the object. Using the same multi-view renders, a segmentation model is prompted with "person.<obj__name>" to produce one mask for the person and one for the object. Visible vertices are projected and labeled per view, then aggregated using scores that combine viewpoint quality (geometric coverage and segmentation reliability) with a boundary-closeness weight for contact regions. Thresholding gives the object label, and geometric filtering cleans the resulting object point cloud.

5. Optionally align a template. If an object template exists, semantic correspondences are established with Zhang et al.'s geometry-aware matcher: the top K_in = 5 LRM renders most similar to the input image are retained, then the top K_pair = 3 template-to-LRM render pairs. Dense bidirectional pixel matches are unprojected to 3D, filtered by cosine similarity above τ_loc = 0.6 and a 2D radius of τ_2D = 15 pixels, and passed through cycle consistency. A weighted Sim(3) Procrustes/Kabsch fit is then iteratively refined using nearest-neighbor constraints to the segmented LRM point cloud, with a constant weight of 10 for each nearest-neighbor pair.

Why This Matters

Impact on research. The paper argues that LRMs can serve as a general geometric prior for interaction reasoning, shifting HOI reconstruction away from 2D reprojection and contact-assumption pipelines. The demonstration that MILO works without templates, without contact annotations, and with interchangeable LRM backbones suggests a more scalable, label-efficient route than methods relying on shape repositories or ground-truth contact, and the strong geometry-inferred contact numbers show reconstructions can substitute for dedicated contact predictors.

Real-world applications:

• Immersive AR/VR: Placing a user and the physical objects they manipulate into a shared 3D scene from a single camera view.
• Robotics and teleoperation: Recovering object pose and human hand/body configuration for imitation learning and manipulation planning.
• Embodied AI and simulation: Generating interaction data for training agents in simulated environments without motion-capture setups.
• Content creation: Turning single photographs into 3D human-object scenes for film, games, and e-commerce visualization.

Industry relevance. The method's compatibility with off-the-shelf LRMs (Hunyuan3D-2.0, SAM3D) and off-the-shelf pose models (HMR2.0, ViTPose, HaMeR) means companies building on improving 3D foundation models can benefit automatically as those backbones improve. The removal of contact and template requirements lowers data-collection costs for product teams, though per-image optimization rather than a single feed-forward pass is stated implicitly by the iterative fitting stages and may matter for latency-sensitive applications.

Future Directions

• Improve the LRM backbone. Performance is explicitly bounded by the quality of the underlying LRM, so better reconstruction models should directly raise MILO's ceiling; the oracle experiment on InterCap (3.28/4.93/3.12) quantifies this headroom.
• Better point-cloud segmentation. The authors state segmentation accuracy hinders performance and expect MILO to benefit from stronger segmentation methods.
• Refine LRM outputs for small objects. Small objects are called out as a specific weakness, along with a need for stronger instance-level reasoning to extract cleaner object meshes.
• Extend beyond single images. The authors propose extending the framework to video and to multi-person/multi-object interaction settings.

Target Audience

Researchers and graduate students in 3D computer vision, especially those working on human-object interaction, monocular human reconstruction, or 3D generative models. It is also useful for practitioners in AR/VR, robotics, and embodied AI who need single-image interaction capture without contact annotations or object template libraries, and for readers tracking how Large Reconstruction Models can be repurposed as geometric priors rather than end products.

Authors’ abstract

Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction procedure, reframing the problem as interpreting the LRM mesh: we segment it into human and object components, fit a parametric body model to the human part, and optionally align an object template to the object part (if such a template is available). MILO achieves strong reconstruction accuracy and outperforms existing baselines across multiple benchmarks and interaction scenarios. Our code is available at https://ac5113.github.io/MILO.