MAMHOI: Factorizing Scene-Aware Human-Object Interaction through Affordances

AI summary

Overview

• Research area: Computer Vision / 3D human motion generation — specifically scene-aware human–object interaction (HOI) synthesis conditioned on language, object geometry, and 3D indoor scenes.
• Technical level: Advanced. The paper assumes familiarity with diffusion models, affordance representations, voxel grids, and text-to-motion benchmarks.
• Scope: A single paper presenting MAMHOI, a factorization that splits scene-aware HOI generation into a scene-to-affordance stage learned from human–scene data and an affordance-to-motion stage learned from scene-free human–object data.

What This Paper Is About

Generating a person picking up and moving an object inside a cluttered 3D room requires two different skills: figuring out where the interaction is physically possible in the scene, and producing how the human and object move together realistically. Supervision for these two skills rarely exists in the same dataset — human–scene datasets show environment-aware motion, and human–object datasets show detailed manipulation, but paired human–object–scene data are scarce. The paper's goal is to learn both skills from their respective data sources and connect them through an explicit intermediate representation rather than requiring joint paired data.

Key Contributions

1. Affordance-mediated factorization. MAMHOI separates scene-level interaction feasibility from detailed human–object motion synthesis, so the scene reasoning component and the motion generator can each be trained on the data source that supervises it best, without paired human–object–scene data.
2. Motion affordance as an explicit interface. The paper defines a motion affordance — the spatial support in which a human–object interaction can feasibly evolve, represented as a sequence-level binary 2D region covering both the human and the manipulated object — and uses it to pass scene constraints to the motion generator without prescribing a single trajectory.
3. A concrete conditioning pipeline. The predicted 2D affordance is lifted to 3D, queried as three local 32×32×32 occupancy voxels around the pelvis, current object position, and object goal, and injected into a diffusion backbone via a ViT-based encoder and cross-attention.
4. Empirical gains in scene feasibility with preserved interaction quality. On a constructed scene-aware test set, MAMHOI reduces human–scene and object–scene penetration relative to CHOIS and InfBaGel while improving object-path adherence and human–object contact, and it is preferred by human raters in a 2AFC study.

Main Findings

• Scene penetration is substantially lower. On the constructed LINGO-scene test set, MAMHOI without any test-time refinement achieves human–scene vertex penetration $\bar{D}^{\mathrm{vtx}}$ of 28.587 (×10⁻³) and frame penetration $\bar{D}^{\mathrm{frm}}$ of 22.009 (×10⁻³), versus 47.337 and 36.232 for CHOIS and 36.478 and 26.315 for InfBaGel.
• Object–scene penetration is the key win. MAMHOI reports object–scene $\bar{D}^{\mathrm{vtx}}$ of 0.403 and $\bar{D}^{\mathrm{frm}}$ of 1.440 (×10⁻³), compared with 0.545 / 2.267 for CHOIS and 3.136 / 7.281 for InfBaGel — the abstract and conclusion single out object–scene penetration as the main improvement.
• Object trajectory adherence is better. $\bar{D}_{\mathrm{path}}^{\mathrm{obj}}$ is 0.1492 for MAMHOI versus 0.1532 for CHOIS, and 0.1537 for CHOIS with test-time SDF guidance; InfBaGel does not report a value in the table.
• Contact quality improves. The fraction of frames with active human–object contact $R^{\mathrm{con}}$ is 94.87% for MAMHOI, against 90.97% (CHOIS), 90.44% (CHOIS + $\mathcal{G}_{\mathrm{sdf}}$), and 92.39% (InfBaGel).
• Motion smoothness is better. Human-motion jerk $J_{\mathrm{H}}^{\mathrm{seq}}$ is 4429.3 for MAMHOI versus 7808.0 for CHOIS, 7906.9 for CHOIS + $\mathcal{G}_{\mathrm{sdf}}$, and 5258.7 for InfBaGel.
• The unguided model beats guided CHOIS. Without $\mathcal{G}{\mathrm{sdf}}$, MAMHOI outperforms CHOIS + $\mathcal{G}{\mathrm{sdf}}$ on every reported human–scene and object–scene measure, on jerk, on object-path error, and on contact — evidence that the gain comes from the learned affordance-conditioned distribution rather than post-hoc geometric correction.
• Test-time SDF guidance is complementary, not a substitute. Adding $\mathcal{G}_{\mathrm{sdf}}$ to MAMHOI gives the lowest human–scene penetration in the paper (25.884 / 19.889 / 13.745 for $\bar{D}^{\mathrm{vtx}}$, $\bar{D}^{\mathrm{frm}}$, $R^{\mathrm{frm}}$) and the highest contact (95.69%), but slightly worsens object–scene frame-averaged penetration (1.440 → 1.498) and path adherence (0.1492 → 0.1507).
• Interaction-only quality is competitive. On the OMOMO validation split with an empty-scene affordance, MAMHOI reaches FID 1.8361, R-precision 0.7764 / 0.9279 / 0.9543 (top-1/2/3), MMD 3.2798, and Diversity 8.6923, improving on CHOIS (2.7892 / 0.7428 / 0.8678 / 0.9062 / 3.9086 / 8.3841) on all four metrics. InfBaGel reports a lower FID (2.2451) than CHOIS but higher than MAMHOI, and the highest diversity (9.2597).
• Removing the ranking loss helps scene-free metrics but is a scene regularizer. MAMHOI without $\mathcal{L}_{\mathrm{rank}}$ reaches FID 0.9403 and MMD 3.1553 — better than the full model — which the authors attribute to the ranking term acting as a scene-conditioning regularizer rather than a motion-fidelity objective. The paper states that scene-aware ablations showing its benefit appear in the Appendix.
• Human raters prefer it. In a 2AFC user study with 136 comparison pairs, 884 valid votes, and 52 responses, MAMHOI was preferred over CHOIS in 86.7% of votes and over CHOIS + $\mathcal{G}{\mathrm{sdf}}$ in 92.3%; MAMHOI + $\mathcal{G}{\mathrm{sdf}}$ was preferred in 81.1% and 84.3% respectively. The authors note that $\mathcal{G}_{\mathrm{sdf}}$ does not improve human preference despite reducing penetration.

Methodology in Plain English

The authors split the problem in two and glue the halves together with a picture of where the action can happen.

Half one — predict the affordance. A scene-understanding model takes a 2D depth map of the scene, an object path supplied by a simple A* planner, and a CLIP text embedding of the instruction. A residual 2D U-Net encodes the depth map; at its bottleneck, scene features act as queries in cross-attention against path and text features as keys and values. The output is a 2D affordance probability map, thresholded to a binary mask. Training targets are built from real scene-aware motion trajectories, using a denoising loss plus a path-weighted binary cross-entropy loss (decay parameter τ = 3, maximum weight 6) and a path-coverage loss that weights the start and end points at ω = 4 and other points at 1, with λ_BCE = 0.1 and λ_path = 0.2.

The interface. The 2D affordance is lifted into 3D using the scene geometry. For each autoregressive motion segment, the system queries three 32×32×32 binary occupancy voxel grids centered on the human pelvis, the current object position, and the object goal, concatenated into a 3×32×32×32 tensor. Because the reference points move as the interaction progresses, the voxels are re-queried every segment.

Half two — generate the motion. A diffusion model predicts clean human–object motion (3D joint positions plus 6D joint rotations for the human, centroid and rotation for the object, in a canonical human-local frame) from noisy motion, sparse conditions (initial state, object waypoints, goal state), a CLIP text embedding, a BPS object-geometry descriptor of size 1024×3, and the affordance. A ViT-based encoder turns the voxels into affordance features that enter the denoiser through cross-attention as keys and values.

Keeping the generator honest about the scene. Beyond standard reconstruction and forward-kinematics losses, the paper adds a signed-distance-field penalty that pushes human joints inside the valid affordance region with a safety margin, and a hard-negative ranking loss that compares a positive motion and an alternative motion with similar start/goal configurations but different intermediate trajectories, requiring the affordance-compatible one to score better by a margin. Optional test-time SDF guidance on a small set of proxy joints is evaluated separately and explicitly excluded from the proposed method.

Data split. The scene-understanding model is trained on LINGO; the motion backbone on OMOMO. Affordance supervision for both is built from relaxed axis-aligned bounding boxes around the corresponding trajectories. Interaction-only evaluation uses the OMOMO validation split (534 paired sequences of 120 frames, 12 movable objects, 2 unseen subjects). Scene-aware evaluation uses a constructed set from 20 LINGO test scenes, with 164 A* object paths and about 820 trials across five movable objects. CHOIS is retrained with 30-frame windows to match the setting; InfBaGel is evaluated with its native pelvis-trajectory constraint and its native Bump-aware Guidance.

Why This Matters

The work offers a template for combining datasets that supervise different parts of a task rather than trying to collect one dataset that supervises everything. Instead of augmenting or jointly fitting heterogeneous paired data (as UniHM and InfBaGel do), it inserts a compact intermediate representation that both sides can agree on. That is a broadly reusable idea for embodied generation, where scene data and manipulation data are collected by different communities with different sensors and conventions. The result that learned affordance conditioning outperforms post-hoc collision correction — and that collision correction does not improve human preference — is also a useful caution for anyone tempted to bolt geometric fixes onto a generator.

Real-world applications:

• Character animation and VFX. Generating plausible "walk over, pick up, carry, place" motion inside a modeled set without hand-authored collision cleanup.
• VR/AR and games. On-demand, physically plausible interaction animation for cluttered virtual rooms, conditioned on a text prompt and a target object location.
• Household robotics and embodied AI. Producing candidate human demonstrations of manipulation in scanned rooms, which can seed imitation-learning data where paired robot–scene–object demonstrations are scarce.
• Ergonomics and interior design simulation. Testing whether a layout affords a task (reach, carry path, placement) before physical prototyping.

Industry relevance: the factorization is directly applicable to pipelines where a scene-reconstruction stack and a motion-capture/manipulation stack are owned by different teams — the affordance tensor is a clean handoff artifact.

Future Directions

• Replace A path planning with learned, task-level planning.* The current setup requires a planner-provided object path, and the affordance model is conditioned on it; end-to-end scene-to-affordance prediction without an external planner is an open question.
• Extend beyond 2D sequence-level affordances. The interface is a binary 2D region per sequence with three local 3D queries; richer 3D or contact-aware affordance representations might capture vertical structure and multi-contact interactions that a top-down region misses.
• Resolve the scene-free fidelity trade-off. The ranking loss hurts FID and MMD on OMOMO (0.9403 vs 1.8361 FID without it) while helping scene-condition sensitivity; a schedule or adaptive weighting that keeps both would be valuable.
• Broaden object and scene coverage. Evaluation covers 12 movable objects on OMOMO and five movable objects per path on the LINGO scenes, and the paper criticizes prior work (HOSIG) for limited object coverage — scaling object diversity and testing on real scanned apartments rather than 20 test scenes is a natural next step.

Target Audience

Researchers and graduate students working on text-conditioned motion generation, human–object interaction synthesis, and human–scene interaction, particularly those interested in diffusion-based generative models and in training on complementary, unpaired data sources. It is also relevant to practitioners in character animation, VR/AR content, and embodied AI who need physically feasible interaction motion in reconstructed 3D environments, and who are willing to engage with affordance representations, voxel occupancy grids, and SDF-based losses.

Authors’ abstract

Generating realistic human-object interactions (HOI) in complex 3D scenes requires two complementary capabilities: reasoning about interaction feasibility in the environment and synthesizing realistic human-object motion. However, supervision for these capabilities is rarely available jointly at scale. Human-scene datasets provide rich information about environment-aware motion, while human-object datasets capture detailed interaction dynamics, yet paired human-object-scene data remain scarce. We present MAMHOI, an affordance-mediated factorization for scene-aware human-object interaction generation. MAMHOI factorizes scene-aware HOI generation through an explicit motion-affordance interface between scene understanding and motion synthesis: a scene-conditioned model first predicts where and how an interaction can be feasibly executed, and an affordance-conditioned HOI model then generates the corresponding human-object motion. This factorization allows scene understanding and interaction dynamics to be learned from complementary sources of supervision without requiring paired human-object-scene data. Experiments in complex indoor environments show that MAMHOI reduces object--scene penetration while better preserving human--object interaction quality, yielding more realistic and physically feasible scene-aware interactions. Project page: https://leimingyuan.github.io/MAMHOI-project-page/