Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling

AI summary

Overview

Research area: Computer vision and robotics — 3D object pose and shape estimation, physics-aware scene reconstruction, and simulation-based (digital twin) world modeling.

Technical level: Advanced. The paper builds on maximum likelihood estimation over SIM(3) transforms, signed distance fields, contact scene graphs, Markov Chain Monte Carlo / rejection sampling, and point-cloud registration (Chamfer distance, ICP). The prose is readable, but the math and the comparison baselines assume familiarity with the 6D pose estimation literature.

Scope (1 sentence): The paper introduces Picasso, a physics-constrained sampling pipeline that retrofits existing single-image object pose and shape estimators so that multi-object scene reconstructions become physically plausible (no interpenetration, no floating objects, no penetration of observed free space), together with a new 10-scene contact-rich dataset and a physics plausibility metric.

What This Paper Is About

Modern 3D reconstruction systems estimate the pose and shape of each object largely in isolation, so even geometrically accurate results — ones that fit the sensor data well — can be physically impossible, such as objects interpenetrating each other or floating in mid-air. This matters because importing such estimates into a simulator produces a digital twin that cannot reliably predict how a contact-rich scene will behave if you act on it. Picasso's goal is to reason holistically over the whole scene, treating physics as hard constraints rather than as extra loss terms, and thereby produce reconstructions that are both accurate and physically valid.

Key Contributions

1. Picasso, a physics-constrained reconstruction pipeline. It builds multi-object scene reconstructions by accounting for geometry, non-penetration, and contact. It uses fast rejection sampling over multi-object interactions, guided by an inferred object contact graph, rather than gradient-based optimization of physics penalties. The four constraint families are inter-object non-penetration, object-environment non-penetration, object-free-space non-penetration, and a contact constraint that discourages floating objects (up to a tolerance δ).

2. The Picasso dataset. A collection of 10 contact-rich real-world static scenes containing 10 everyday objects, with ground-truth pose and shape annotations. Each scene uses 10 images, and scenes contain between two and eight objects. Contact types covered include point contacts (e.g., mugs supported by a rack), line contacts (e.g., a spatula resting on a pan), and flush contacts (e.g., stacked Jenga blocks).

3. A physical plausibility metric, the Scene Plausibility Score (SPS). SPS places the estimated scene as a digital twin into a simulator, runs a short rollout of T = 20 steps at a 1/240 s timestep, and measures kinetic energy with a nominal mass of 1 kg and inertia set to the identity matrix. Translational and rotational kinetic energies are thresholded at a maximum of 10 kg·m²/s² in all experiments. The code, dataset, and benchmark are open-sourced.

4. Extensive evaluation. Picasso is evaluated on the new Picasso dataset and on YCB-V, and is retrofitted onto CRISP and SAM3D. The paper reports that it outperforms the state of the art on pose accuracy and physical plausibility, and that results are more aligned with human intuition.

Main Findings

• Picasso closes the sim-to-real gap for CRISP. CRISP-Real (trained on real YCB-V training images) outperforms CRISP-Syn (trained on 4,200 synthetic images, 200 per object, rendered with BlenderProc), indicating a large sim-to-real gap. Adding Picasso makes CRISP-Syn the top-performing method, surpassing CRISP-Real and outperforming local refinements such as gradient descent and GICP.

• Pose accuracy on YCB-V (Table I, ADD-S in meters, lower is better). CRISP-Syn+Picasso achieves a mean of 0.008 and median of 0.003, with ADD-S AUC of 0.52 at 1 cm, 0.69 at 2 cm, and 0.77 at 3 cm. CRISP-Real+Picasso is best overall at 0.008 mean, 0.003 median, and AUC 0.53 / 0.70 / 0.78. For comparison, CRISP-Real alone scores 0.009 mean, 0.004 median, and 0.44 / 0.58 / 0.75; CosyPose scores 0.010 / 0.007 and 0.30 / 0.56 / 0.68; BundleSDF scores 0.014 / 0.012 and 0.14 / 0.37 / 0.55; GDRNet++ scores 0.013 / 0.011 and 0.22 / 0.43 / 0.58; CRISP-Syn and CRISP-Syn+GD both score 0.015 mean with AUC 0.21 / 0.41 / 0.55.

• Physics metrics and speed on YCB-V (Table II). CRISP-Syn+Picasso achieves SPS 8.71, NPS 3.99 mm, and observable correctness of 55%, with a runtime of 0.35 s. CRISP-Real+Picasso achieves SPS 8.77, NPS 3.95 mm, and OC 55%, with runtime 0.33 s. Baselines: CRISP-Real SPS 9.05, NPS 4.57, OC 36%; CRISP-Syn SPS 9.36, NPS 6.38, OC 15%; CRISP-Syn+GICP SPS 9.56, NPS 4.70, OC 42%, runtime 0.03 s. Runtime is reported as "/" (not applicable) for the two unaugmented CRISP variants.

• Picasso improves SAM3D. On the YCB-V dataset from the BOP19 challenge (900 frames) and on the Picasso dataset, Picasso significantly improves SAM3D on both a geometric metric (ADD-S) and physical metrics (NPS and SPS). Picasso also outperforms an ablation without physics constraints (SAM3D+Picasso w/o PC, which only samples from SIM(3) to minimize the objective), but that benefit is described as more modest because physics plausibility is only checked on a small buffer of candidates ranked by Chamfer loss.

• The reported numbers for the SAM3D comparison table (Table III) are not visible in the provided content. The table caption states it compares methods across 12 YCB-Video trajectories using ADD-S (mm), SPS, and NPS (mm), and its numeric cells are truncated.

• Physics constraints disambiguate ambiguous poses. The paper illustrates a case where the loss landscape over SE(3) for a bowl contains two global minima due to partial point-cloud observations; only one satisfies the physics constraints, and the constraints prune the feasible set to identify the correct pose.

• Rejection sampling settings. Picasso applies rejection sampling to the top B = 16 candidates ranked by Chamfer loss; if all B violate the constraints, it returns the best-scoring pose. A Geman-McClure robust loss with δ = 0.05 m is applied to the Chamfer distance to mitigate segmentation artifacts, shape prediction errors, and depth sensor noise. Physics checks use libigl for object-object penetration and contact, and a voxel-based depth renderer for object-free-space penetration.

Methodology in Plain English

The problem is framed as maximum likelihood estimation: given an RGB image, depth map, and per-object masks, find each object's scale, rotation, translation (a SIM(3) transform) and shape that best explain the measurements while remaining physically plausible.

The measurement likelihood is factored into a shape term and a pose term. The shape term is delegated to an off-the-shelf shape estimator (SAM3D or CRISP), which may return either a single estimate or a discrete set of candidates. The pose term, under independent noise, becomes a sum of per-object distances between measured 3D points transformed into the object frame and the object's surface points — concretely the Chamfer distance.

Physical plausibility is encoded as four analytic constraints expressed through signed distance fields: no object-object interpenetration, no penetration of the environment (a planar surface such as a table, estimated with 3-point RANSAC from depth), no penetration of observed free space, and contact — every object must touch at least one other object or the environment within tolerance δ. Free-space checking renders the object as a depth map and compares directly against the observed depth.

Solving this joint problem over N similarity transforms is highly nonconvex and, if sampled naively, high-dimensional. So the authors split it using a contact scene graph: a graph whose nodes are the N+1 rigid bodies (including the environment) and whose edges denote physical contact. The graph topology is inferred with a vision-language model rather than by point-cloud proximity, because proximity checks were found sensitive to noise and occlusions. The CSG is then approximated as a directed acyclic graph using breadth-first search with the environment as root, enabling a greedy rollout in which one object is estimated at a time, conditioned on already-fixed parents. This reduces inference calls from O(N) for the full CSG to exactly N on the DAG, and decouples the problem into N independent subproblems. The authors acknowledge this is an approximation that is not guaranteed to recover the global maximum-likelihood assignment, though they observe excellent accuracy in practice.

Each subproblem is solved by coarse-to-fine rejection sampling over SIM(3), separating scale from SE(3): a broad set of scale hypotheses is refined progressively, and for each scale hypothesis an SE(3) registration subroutine samples translation and rotation coarse-to-fine, retaining top candidates by loss and resampling locally around them. Candidates violating the physics constraints are simply rejected, which avoids reshaping the objective with competing penalty terms and encourages global exploration of the state space. Initialization uses the mean of the masked depth point cloud (after 5% outlier thresholding) for translation, and a comparison of mean masked depth to mean rendered depth from the network's estimate for scale.

The dataset was captured with an iPad's built-in LiDAR and camera at 1920 × 1440 resolution and 60 Hz. Masks were produced automatically with SAM2 and then verified and corrected by human annotators; object models were scanned and refined in Blender; 3D poses were obtained by solving PnP from labeled 2D keypoints matched to keypoints on the scanned models; and raw depth maps were refined by rendering the object models under the annotated poses.

Why This Matters

Impact on research. The paper argues for holistic, multi-object reasoning in pose and shape estimation rather than per-object reasoning, and it offers a concrete alternative to physics-guided loss optimization (prone to local minima) and differentiable-simulator supervision (brittle due to modeling inaccuracies and numerical noise). It also contributes a benchmark and metric for an aspect of reconstruction quality that geometric metrics alone do not capture, and shows a simulation-trained model can be corrected to beat a real-data-trained one.

Real-world applications.

• Simulation-based planning and control of contact-rich manipulation, where a robot needs a digital twin whose behavior under intervention is predictive.
• Robotic manipulation in cluttered domestic scenes, such as unsticking stacked plates or pans in a sink, or reasoning about which Jenga block can be removed safely.
• Digital twin construction from a single RGB-D observation, for inspection, inventory, or rearrangement tasks.
• Benchmarking and regression testing of pose and shape estimators, using the Picasso dataset and SPS to flag physically implausible outputs that pass geometric metrics.

Industry relevance. The pipeline is described as a post-hoc corrector that retrofits existing estimators rather than replacing them, and it runs in under a second per scene (0.33–0.35 s in the reported YCB-V experiments), which matters for practical deployment. The reporting of a nominal-mass, nominal-inertia kinetic energy metric gives industry a comparable, thresholded number for scene validity. Code and dataset are released at the MIT-SPARK GitHub repository.

Future Directions

• Scaling beyond the greedy DAG approximation. The authors state the DAG rollout is not guaranteed to recover the global maximum-likelihood assignment in the full constrained problem, leaving open whether better graph decompositions or exact inference can close that gap and whether the approach scales to larger and cyclic contact structures (the paper notes Jenga blocks can form a ring of n nodes).
• Broadening the constraint set. The current formulation restricts itself to static scenes and configuration-space (holonomic) constraints — non-penetration, contact proximity, and a planar environment. Extending to non-planar environments, non-rigid objects, or dynamic scenes is not addressed.
• Strengthening the physics-checking stage. The ablation against sampling without physics constraints showed only a modest benefit, which the authors attribute to checking physics plausibility on a small buffer of candidates (B = 16) ranked by Chamfer loss. Improving candidate generation or constraint checking could yield larger gains.
• Understanding simulator-induced limitations. The SPS metric relies on a simulator (PyBullet in this work), and the authors note potential inaccuracies in the simulator's physical modeling, which is why kinetic energies are thresholded at 10 kg·m²/s². How much the metric reflects true physical plausibility versus simulator artifacts remains an open question.

Target Audience

Robotics and computer vision researchers working on 6D object pose and shape estimation, scene understanding, and physics-aware perception; engineers building digital twins or simulator-based planning and control pipelines; and practitioners who want to improve an existing single-image pose/shape estimator with a lightweight, sampling-based correctness layer. Readers need comfort with optimization, probability, and 3D geometry to follow the derivations, though the conceptual message is accessible to a broader audience.

Authors’ abstract

In the presence of occlusions and measurement noise, geometrically accurate scene reconstructions -- which fit the sensor data -- can still be physically incorrect. For instance, when estimating the poses and shapes of objects in the scene and importing the resulting estimates into a simulator, small errors might translate to implausible configurations including object interpenetration or unstable equilibrium. This makes it difficult to predict the dynamic behavior of the scene using a digital twin, an important step in simulation-based planning and control of contact-rich behaviors. In this paper, we posit that object pose and shape estimation requires reasoning holistically over the scene (instead of reasoning about each object in isolation), accounting for object interactions and physical plausibility. Towards this goal, our first contribution is Picasso, a physics-constrained reconstruction pipeline that builds multi-object scene reconstructions by considering geometry, non-penetration, and physics. Picasso relies on a fast rejection sampling method that reasons over multi-object interactions, leveraging an inferred object contact graph to guide samples. Second, we propose the Picasso dataset, a collection of 10 contact-rich real-world scenes with ground truth annotations, as well as a metric to quantify physical plausibility, which we open-source as part of our benchmark. Finally, we provide an extensive evaluation of Picasso on our newly introduced dataset and on the YCB-V dataset, and show it largely outperforms the state of the art while providing reconstructions that are both physically plausible and more aligned with human intuition.