SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models

AI summary

Overview

• Research area: Robotics / embodied AI — specifically robotic manipulation planning with Vision-Language Models (VLMs), real-to-simulation reconstruction, and test-time reasoning.
• Technical level: Intermediate. The paper assumes some familiarity with VLMs, physics simulators, and robot action representations, but its central idea is conceptually simple.
• Scope: A test-time, zero-shot framework that builds a physics simulator from a single RGB-D image and uses simulated rollouts as context for a VLM to iteratively propose and refine robot manipulation actions.

What This Paper Is About

Vision-Language Models are good at describing scenes and tasks in language but have no grounded understanding of physical dynamics, because they are trained on static internet-scale images and text containing no causal interactions or action-conditioned change. This means they can say what to do but cannot reliably predict how an action will unfold physically, which hurts fine-grained manipulation tasks where small contact or timing variations change the outcome entirely. SIMPACT addresses this by generating a physics simulation from a single RGB-D observation and feeding simulated rollouts back into the VLM at test time, so the model reasons about dynamics without any additional training.

Key Contributions

1. A test-time, zero-shot framework that lets VLMs plan physics-aware embodied actions, requiring no task-specific training.
2. An automated pipeline that builds multi-physics simulations from a single RGB-D observation using pretrained visual foundation models (segmentation, image-to-3D, pose estimation) plus a VLM that selects simulators and infers physical parameters. Both rigid (mesh-based, MuJoCo) and deformable (particle-based) objects are supported.
3. An in-context learning approach for robot action generation in which physics simulation rollouts serve as context, enabling a new form of test-time reasoning in robotics.
4. Empirical validation on seven real-world rigid-body and deformable manipulation tasks, with success rates substantially above general-purpose baselines, plus ablations isolating each component.

Main Findings

• Outperforms all baselines across the board: Over 10 trials per task, SIMPACT achieved 80% (non-toppling push), 60% (bowl stacking), 40% (pivoting), 90% (shape rope), 80% (shape dough), 80% (avoid obstacle), and 70% (sweeping).
• The VLA baseline fails entirely: π_0.5 scored 0% on all seven tasks. The authors attribute this to poor generalization to out-of-domain tasks, even though it sometimes generates actions that approach the target object.
• Geometry-augmented VLM baselines are weak on precision: VoxPoser scored 20% on bowl stacking and 20% on sweeping, and 0% on the other five tasks. MOKA scored 20% on shape rope, 10% on bowl stacking, and 0% on the rest. Both fail when precise contact placement matters.
• VLM-guided sampling matters most: Removing the VLM sampler (replacing it with Gaussian sampling over gripper pose deltas, with 5x more samples) dropped results to 0%, 10%, 0%, 0%, 0%, 0%, 20%.
• Simulation rollouts are essential for physical grounding: Removing the simulation rollout context dropped results to 20%, 0%, 0%, 30%, 30%, 0%, 20%, with the largest damage on bowl stacking, pivoting, and obstacle avoidance. This variant still beat the baselines, which the authors credit to hierarchical action sampling.
• The VLM optimizer adds non-local reasoning: Removing iterative refinement dropped results to 30%, 50%, 30%, 40%, 80%, 20%, 70%, with the biggest losses on non-toppling push, shape rope, and avoid obstacle.
• More samples help only up to a point: On three tasks, 3 samples gave 50% / 50% / 40%, 10 samples gave 80% / 60% / 90%, and 20 samples gave 90% / 20% / 80% (non-toppling push / bowl stacking / shape rope). Performance on bowl stacking and shape rope dropped from 10 to 20 samples, which the authors attribute to longer contexts reducing VLM reasoning efficacy.
• Simulation predicts real-world success well: A supplementary experiment reports 89% agreement between simulation and real-world performance.
• Failures decompose into three categories: perception failures (single-view 3D reconstruction errors), planning failures (most frequent, especially in pivoting), and execution failures (kinematic or dynamic sim-to-real discrepancies).

Methodology in Plain English

The pipeline has two stages.

Stage 1 — build a simulator from one image. Given an RGB-D image and a task instruction, a VLM first names the objects that matter and decides whether each is rigid or deformable. GroundedSAM2 segments each object. For rigid objects, an image-to-3D model reconstructs a full triangle mesh, which is then centered and rescaled to match the real object's bounding box, and FoundationPose estimates its 6DoF pose. For deformable objects, the segmented mask is back-projected through the depth image to get surface points, and particles are uniformly sampled in the volume between the object surface and the table. The VLM also chooses the physics engine — MuJoCo for rigid bodies, a projective dynamics variant for stiff deformables, and the Material Point Method for soft objects — and predicts physical parameters such as mass, friction, center of mass, elasticity, and plasticity through a question-answering procedure.

Stage 2 — plan with simulated rollouts in the loop. The VLM does not emit raw 6-DoF poses directly, because those are hard for language models to reason about. Instead it emits compact symbolic actions — PUSH, LIFT, DESCEND, GRASP, RELEASE, ROTATE, MOVE — each with continuous parameters, and a deterministic Action2Pose mapping converts them into end-effector trajectories. The planner samples K=10 initial action proposals, rolls each out in the simulator, then passes rendered simulator images plus numerical actions and states back to the VLM, which proposes an improved sequence. The VLM also judges whether the final simulated state satisfies the task. The loop runs until success or K_max = 13 (up to 3 optimization iterations), and the winning action sequence is executed on the real robot. The default VLM is Google Gemini 2.5 Pro; the projective dynamics solver is implemented in PyTorch and the MPM simulator in Warp. Hardware is a Franka Research 3 arm with a parallel-jaw gripper (a custom 3D-printed end effector for the Play-Doh task) and a single calibrated Intel RealSense D435i camera.

Why This Matters

Research impact. The paper argues that the missing ingredient for VLM-based manipulation is not more action-labeled training data but test-time access to a physically grounded world model. It connects model-based control ideas with foundation-model perception, and it shows that simulation can act as in-context evidence rather than merely as a reward signal (as in the related Prompting-with-the-Future approach, whose CEM-based variant the authors report achieves near-zero success). It also advances real-to-sim construction by using a single RGB-D view rather than full digital twins or cousin-creation pipelines.

Real-world applications.

• Warehouse and logistics manipulation, where pushing, stacking, and sweeping objects without toppling are routine but difficult to script.
• Food handling and baking, where deformable materials such as dough and rope-like items must be shaped to geometric targets.
• Household robotics, where a robot must avoid obstacles while moving objects in cluttered, unseen scenes.
• Rapid task onboarding in factories, where a new instruction can be attempted zero-shot with a quick scene scan instead of a data-collection campaign.

Industry relevance. The framework requires no task-specific training and reuses off-the-shelf foundation models, which lowers the barrier for deploying general-purpose arms. The reported 89% simulation-to-real agreement is the kind of number that matters commercially, because it indicates when a simulated plan can be trusted before it is executed on expensive hardware, and it points toward simulation as a verification layer.

Future Directions

• Better single-view 3D reconstruction. The authors note that rigid-body simulation quality depends heavily on image-to-3D models that struggle with occlusions, and suggest inpainting, generative 3D completion, or articulated object reconstruction as remedies.
• System identification from real interaction. Physical parameters are currently predicted by VLM prompting, so inaccuracies cause sim-to-real gaps; refining estimates with real-world interaction data is a natural next step.
• Closing the loop at execution time. The current system is open-loop and vulnerable to compounding errors. An optional replanning mechanism that updates the simulator from real-world feedback is included in the supplement, and tighter real-time feedback integration is left to future work.
• Managing context length in the optimizer. The drop in bowl stacking and shape rope performance when moving from 10 to 20 samples suggests that pre-selecting informative rollouts, rather than simply adding more, is an open problem.

Target Audience

Robotics and embodied-AI researchers working on manipulation, real-to-sim transfer, and VLM-based planning; engineers evaluating zero-shot policies for general-purpose robot arms; and graduate students who want a concrete example of combining foundation models with physics simulation at test time. Readers already familiar with VLAs, VoxPoser-style affordance representations, and physics engines will get the most from the method and ablation details, though the core idea is accessible to anyone who understands that language models need physical grounding to act reliably.

Authors’ abstract

Vision-Language Models (VLMs) exhibit remarkable common-sense and semantic reasoning capabilities. However, they lack a grounded understanding of physical dynamics. This limitation arises from training VLMs on static internet-scale visual-language data that contain no causal interactions or action-conditioned changes. Consequently, it remains challenging to leverage VLMs for fine-grained robotic manipulation tasks that require physical understanding, reasoning, and corresponding action planning. To overcome this, we present SIMPACT, a test-time, SIMulation-enabled ACTion Planning framework that equips VLMs with physical reasoning through simulation-in-the-loop world modeling, without requiring any additional training. From a single RGB-D observation, SIMPACT efficiently constructs physics simulations, enabling the VLM to propose informed actions, observe simulated rollouts, and iteratively refine its reasoning. By integrating language reasoning with physics prediction, our simulation-enabled VLM can understand contact dynamics and action outcomes in a physically grounded way. Our method demonstrates state-of-the-art performance on five challenging, real-world rigid-body and deformable manipulation tasks that require fine-grained physical reasoning, outperforming existing general-purpose robotic manipulation models. Our results demonstrate that embedding physics understanding via efficient simulation into VLM reasoning at test time offers a promising path towards generalizable embodied intelligence. Project webpage can be found at https://simpact-bot.github.io