CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives

AI summary

Overview

Research area: Computer vision and robotics — specifically real-to-sim reconstruction, monocular human motion recovery, human-scene interaction, and reinforcement-learning-based physics simulation.

Technical level: Advanced. The paper assumes familiarity with SLAM-based 4D reconstruction, SMPL body models, vision-language models, convex geometry for collision detection, and on-policy RL for motion tracking (PPO, GAE, Isaac Gym).

Scope: A single paper introducing CRISP, a pipeline that converts a casually captured monocular RGB video of a person interacting with a static scene into compact planar scene primitives plus physics-validated human motion, then validates both by training a humanoid controller in simulation.

What This Paper Is About

Given an ordinary handheld video of a person sitting, climbing stairs, or moving through a parkour-like space, the authors want to recover both the 3D human motion and the 3D scene in a form that a physics simulator can actually use. The core problem is that typical reconstruction pipelines output noisy, non-watertight point clouds or oversized meshes, and even small geometric artifacts cause simulated humanoid characters to get stuck, bounce, or fall. The goal is to produce "simulation-ready" humans, scenes, and contacts from that single video.

Key Contributions

1. Planar primitive fitting as the scene representation. Rather than fusing points into a TSDF and meshing with Marching Cubes, CRISP clusters 3D point clouds into roughly 50 compact, convex, watertight planar primitives, which are cheaper to simulate and regularize away low-level noise.
2. Contact-guided scene completion. Using a vision-language model (InteractVLM) to predict per-SMPL-vertex contact, plus temporal-kinematic filtering to suppress false positives, CRISP hallucinates occluded support surfaces (chair seats, stair treads, platforms) that the human body hides from the camera.
3. Physics in the loop. The reconstructed human and scene drive a fully-constrained motion-tracking policy trained with RL, which both validates and refines the reconstruction rather than treating geometry and motion as independent outputs.
4. Replacement of the depth estimator in MegaSAM with MoGe to produce a scale-invariant dense point cloud, with metric scale recovered from the known size of a typical human.

Main Findings

• Failure rate reduction: CRISP reduces motion-tracking failure rates from 55.2% to 6.9% on human-centric video benchmarks (EMDB, PROX). Figure 1 states that CRISP lowers motion-tracking failure rates by 8 times compared to prior art.
• RL success and throughput: CRISP's planar primitive representation reaches 93.1% RL success with 23K FPS, versus VideoMimic's 44.8% success at 16K FPS — the paper reports this as a 43% faster RL simulation throughput relative to dense-mesh approaches.
• Geometry ablations (Table 1): Progressively better geometry improves simulation. VideoMimic scores 0.337 bidirectional Chamfer and 0.311 one-way Chamfer with 0.928 non-penetration; Ours (TSDF) reaches 75.9% success with 0.178/0.222 Chamfer; Ours (NKSR) reaches 79.3% success with 0.163/0.187; Ours (Planar) reaches 93.1% success with 0.187/0.174 and the best non-penetration score of 0.947. The planar representation has a slightly worse bidirectional Chamfer than NKSR but the lowest one-way (Recon→GT) error.
• Why the Chamfer gap is acceptable: The authors argue the bidirectional term penalizes missing fine-grained, non-contact structures, which are harmless in simulation, whereas extra noisy geometry destabilizes policy rollouts — so the one-sided Recon→GT error is the more meaningful metric here.
• Global human motion (Table 2, EMDB): Without RL, CRISP reports WA-MPJPE₁₀₀ of 78.16 and W-MPJPE₁₀₀ of 179.84, compared to GVHMR (74.80/200.71), TRAM (83.61/249.50), WHAM (98.45/267.53), and VideoMimic (110.64/521.09). With RL refinement, CRISP reaches 70.60/175.93, the best overall in the table.
• Trajectory stability: CRISP with RL reports root translational error of 1.90 and jitter of 8.14, versus VideoMimic with RL at 3.00 and 8.34 and without RL at 2.12 and 9.29.
• Contact ablation (Table 3, no RL-refine, PROX): Enabling contact guidance keeps success at 90.9% but improves Chamfer from 0.193 to 0.187 (two-way), 0.211 to 0.199 (GT→Recon), and 0.175 to 0.173 (Recon→GT). Success does not increase on PROX because most sequences involve sitting; when the seat is missing, the motion simply switches to a squatting pose. VideoMimic scores 27.3% success with 0.337 two-way Chamfer in the same table.
• Qualitative failure modes of the baseline: VideoMimic's reconstructions cause agents to suffer penetrations and get stuck in "ghost surfaces," bounce off protruding surfaces, suffer contact errors from over-smoothed structures, collide with scene artifacts, get trapped by bumpy terrains, and get trapped in local dents (Figure 4).
• Generalization: CRISP is validated on in-the-wild videos including casually captured videos, Internet videos, and Sora-generated videos.
• Stated limitation: Planar decomposition can leave small gaps between neighboring primitives, so reconstructions may appear visually incomplete; the authors report these gaps do not affect simulation quality because key support and contact surfaces are always modeled. Appendix C also notes that contact-augmented planar primitives are occasionally inaccurate because the paradigm depends on HMR and contact estimation accuracy.

Methodology in Plain English

Step 1 — Recover cameras, depth, and a point cloud. The pipeline starts with MegaSAM to get camera intrinsics, per-frame camera poses, and dense depth from the monocular video. The authors swap MegaSAM's depth estimator for MoGe to get a scale-invariant dense point cloud. GVHMR produces an SMPL mesh in camera space, which is lifted to world frame using the estimated camera poses so human, scene, and camera share one coordinate system. Because MegaSAM only recovers geometry up to an unknown scale, the authors use the known size of a human to rescale the point cloud so its human depth matches the depth of the GVHMR mesh.

Step 2 — Fit planar primitives. Physics engines such as Isaac Gym need meshes for collision detection, and point clouds meshed through TSDF fusion produce hundreds of thousands of triangles that are noisy, oversmoothed in places, and artifact-ridden elsewhere. Instead, the authors exploit a planar-world assumption: sitting, lying down, parkour, and stair climbing are largely interactions with planar surfaces. They compute normals from per-frame pointmaps via finite differences, run K-means on the normal maps to get candidate planar segments, use DBSCAN on the 3D points inside each segment to split spatially, and then merge segments across frames using similar planar fits plus sufficient optical flow correspondences. RANSAC fits a plane to each merged region, from which they define a planar cuboid with a default thickness of 0.05 m. The result is on the order of 50 primitives per scene, with no per-scene optimization.

Step 3 — Use contact to fill in what the person is hiding. InteractVLM predicts a binary contact mask over SMPL vertices, but applied naively to video frames it over-predicts "near-contact" false positives. The authors apply non-maximum suppression across time: they keep only predictions with consistently high confidence for L frames and select the frame t* with the smallest amount of human motion within that window. Those contact points are then used to hallucinate support geometry (with a minimum cuboid thickness clamp of 0.05 m), for example reconstructing the occluded seat of a chair that the person is sitting on.

Step 4 — Close the loop with reinforcement learning. A fully-constrained motion-tracking policy is trained to imitate the reconstructed motion, following Peng et al. (2018) for the tracking formulation and MaskedMimic for the observation, action, and reward design. Actions are desired joint targets for a PD controller, modeled as a multivariate Gaussian with fixed diagonal covariance (σ_π = 0.055). The reward exponentiates errors in position, rotation, linear velocity, angular velocity, and root height, plus an energy penalty for smoothness. The policy is a transformer encoder, the critic is an MLP, and training uses Reference State Initialization (10% of episodes start at the first frame, otherwise uniformly along the trajectory) and Early Termination (any joint deviating more than 0.5 m from the reference in world coordinates ends the episode). Simulation runs in Isaac Gym at 120 Hz, policies operate at 30 Hz, and optimization uses PPO with GAE. A separate policy is trained per motion clip. Importantly, Appendix C clarifies that RL does not directly modify the reconstructed scene primitives — it uses them as collision geometry and tests whether the motion can be executed stably.

Why This Matters

Impact on research. The paper argues that "true human understanding is physical" — a foot isn't just stepping down, it's making contact with a support surface. It offers a counterpoint to the dominant paradigm of stacking data-driven priors (feed-forward HMR, geometric foundation models) with no physics in the loop, and shows empirically that physics can improve, not just validate, reconstruction quality. It also reports a system-to-system comparison against the concurrent VideoMimic framework under a shared tracking and RL training pipeline.

Real-world applications (each grounded in the paper's stated motivations):

• Embodied AI and robotics, where scalable real-to-sim asset generation supports policy learning from ordinary video.
• Physically-plausible character animation, producing motion that obeys contact and non-penetration constraints.
• AR/VR, where reconstructed scenes and humans must be simulatable to support interaction.
• Training humanoids from in-the-wild Internet video, including Sora-generated clips, without manual scene annotation.

Industry relevance. Simulation throughput is a practical bottleneck: collision detection is cheapest when geometry is well approximated by convex primitives, and complicated scene geometry becomes expensive quickly. CRISP's ~50 convex primitives per scene and reported 43% faster throughput directly target that cost. The authors state that code and interactive demos are available at crisp-real2sim.github.io/CRISP-Real2Sim, and that code and data will be open-sourced upon acceptance.

Future Directions

• Closing the remaining geometric gaps. The authors note that planar decomposition leaves small gaps between neighboring primitives, making reconstructions look visually incomplete. A follow-up could merge or heal adjacent primitives without reintroducing noisy collision geometry.
• Making contact guidance more robust. Appendix C identifies a failure pattern: the contact-completion paradigm depends on the accuracy of both HMR and the off-the-shelf contact prediction module, and the estimated contact-augmented primitives are occasionally inaccurate. Better contact estimation, or feedback from simulation back into the contact predictions, is an open problem.
• Extending physics to non-planar and deformable interaction surfaces. The planar-world assumption covers sitting, lying down, parkour, and stair climbing, but the authors do not claim it covers curved, articulated, or deformable scenes.
• Revisiting the direction of the physics loop. Appendix C states that RL currently uses the reconstructed primitives as fixed collision geometry rather than modifying them. Letting simulation feedback edit the scene primitives themselves would be a natural next step.

Target Audience

Researchers and engineers working on real-to-sim transfer, monocular human-scene reconstruction, humanoid control, and physically-based character animation. It is most useful to readers who already understand SMPL-based human mesh recovery, visual SLAM, and RL for motion tracking, and who are deciding between dense-mesh, TSDF, NKSR-style, and primitive-based scene representations for downstream simulation. Readers from robotics and AR/VR who need simulation-ready assets from ordinary video are the likely secondary audience.

Authors’ abstract

We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video. Prior work on joint human-scene reconstruction relies on data-driven priors and joint optimization with no physics in the loop, or recovers noisy geometry with artifacts that cause motion tracking policies with scene interactions to fail. In contrast, our key insight is to recover convex, clean, and simulation-ready geometry by fitting planar primitives to a point cloud reconstruction of the scene, via a simple clustering pipeline over depth, normals, and flow. To reconstruct scene geometry that might be occluded during interactions, we make use of human-scene contact modeling (e.g., we use human posture to reconstruct the occluded seat of a chair). Finally, we ensure that human and scene reconstructions are physically-plausible by using them to drive a humanoid controller via reinforcement learning. Our approach reduces motion tracking failure rates from 55.2\% to 6.9\% on human-centric video benchmarks (EMDB, PROX), while delivering a 43\% faster RL simulation throughput. We further validate it on in-the-wild videos including casually-captured videos, Internet videos, and even Sora-generated videos. This demonstrates CRISP's ability to generate physically-valid human motion and interaction environments at scale, greatly advancing real-to-sim applications for robotics and AR/VR.