DeformSmith: Physics Harness-Guided Hierarchical Generation of Deformable Assets for Robot Manipulation

AI summary

Overview

Research area: Robotics — physics-grounded generative asset creation, deformable object simulation, and robot manipulation data synthesis.

Technical level: Advanced. The paper assumes familiarity with the Material Point Method (MPM), continuum material models (neo-Hookean), 3D Gaussian Splatting, convex decomposition for collision proxies, and LLM-agent pipelines that call simulators.

Scope: DeformSmith is a framework that turns text or a single image into an interactive deformable asset — geometry, physical model, material behavior, and robot-ready interaction data — by progressive agentic construction under a shared simulation-based evaluation harness.

What This Paper Is About

Making a deformable object usable in robot simulation requires more than a good-looking 3D shape: it needs mass, volume, stiffness, damping, contact properties, and a verified ability to be grasped, carried, and released. Text and images provide almost no direct evidence about how an object deforms or responds to contact, so any material parameters inferred from them are only initial estimates. The paper's goal is a fully automated pipeline that builds such assets layer by layer, tests each decision through simulation and simulated robot manipulation, and revises the asset until it is physically credible and ready for manipulation.

Key Contributions

1. A hierarchical agentic generation framework (DeformSmith) that converts text or a single image into interactive, physically credible deformable assets, progressively constructing and validating coupled geometry, physical models, material behavior, and robot-interaction readiness across four layers (L0–L3).

2. A shared physics-grounded harness that couples proposal, simulation-based evaluation, and revision across the hierarchy, so that refinement stays consistent with requirements already established at earlier layers as the asset evolves.

3. Robot manipulation inside the generation loop: simulated pick-and-place feedback validates and refines assets beyond basic physical probes, and the system also emits replayable simulated interaction data (robot commands, particle states, contact observations, task outcomes), retaining failed attempts as labeled diagnostic evidence.

4. Quantitative and human evaluation against state-of-the-art image-to-3D/4D baselines (PhysGen3D, PhysGM, PhysX-Omni), plus ablations isolating the hierarchy, the harness, and manipulation-guided refinement.

Main Findings

• DeformSmith leads on all three rated criteria. Mean GPT-6 Astra ratings (0–1) were 0.70 physical realism, 0.58 photorealism, and 0.82 semantic consistency, versus PhysGen3D (0.46 / 0.34 / 0.80), PhysGM (0.39 / 0.27 / 0.68), and PhysX-Omni (0.41 / 0.32 / 0.69). Physical realism and photorealism each exceed the strongest baseline, PhysGen3D, by 0.24; the semantic-consistency gain is smaller (0.82 versus 0.80).

• Blinded human comparisons favor DeformSmith consistently. With 40 participants rating anonymized, side-randomized pairs, win rates were 68% / 88% / 63% against PhysGen3D, 73% / 95% / 71% against PhysGM, and 75% / 96% / 76% against PhysX-Omni, for physical plausibility, visual quality, and semantic consistency respectively (ties excluded).

• Comparisons used a fixed interaction protocol. All methods received matched input images and the same interaction description: a five-second gravity drop onto a horizontal table from a clearance of one quarter of object height, recorded at 30 fps from two views, with matched physical duration and playback speed. Inputs to the rating model omitted method labels.

• Baselines fail mainly after contact, not before. In the qualitative comparison (turtle, whale, seal, rugby ball), PhysGen3D produced recognizable initial assets but showed pronounced flattening or collapse after the drop; PhysGM showed severe spreading and surface disruption in the plush cases; PhysX-Omni exhibited incomplete geometry, including a fragmented seal. The paper argues that recognizable initial appearance is insufficient if the object does not stay coherent while deforming.

• Hierarchical construction helps delivery and physical validity at comparable cost. Hierarchical versus Flat: asset delivery 93% versus 83%, independent physics-test success 87% versus 80%, at 13 versus 13.7 simulation calls per asset. Both variants used the same inputs, initial geometry, models, feedback, and computation limits, with missing assets counted as failures.

• The shared harness sharply improves material targeting. Harness versus One-shot: material-target satisfaction 73% versus 40%, hard failures 7% versus 17%, at 14 versus 0 construction simulation calls (the zero excludes the shared final evaluation). Target satisfaction and hard failure are not complementary rates — a physically valid asset can still miss the requested material behavior.

• Manipulation feedback improves both task and material outcomes. Full versus No feedback: pick-and-place task success 40% versus 67%, material pass rate 67% versus 83%, joint success 27% versus 57%. Joint success requires both a successful robot trial and passing the independent material tests. Task and joint rates are measured over robot trials; material pass rate is over assets. Six assets with five held-out conditions each were used, starting from the same L2 assets, with asset and action choices frozen before held-out evaluation.

• The evaluation measures perceived quality, not parameter accuracy. The paper explicitly states the results assess perceived quality under the tested interaction without establishing material-parameter accuracy, and that harness gains reflect both structured feedback and additional simulation effort.

Methodology in Plain English

Start with a description or a picture. Text input is first converted into a reference image; image input is used directly. The object is segmented, and its mesh plus Gaussian appearance are reconstructed. Camera and metric geometry are estimated for alignment, and the mesh is decomposed into convex pieces to serve as a collision proxy. Mesh, collision proxy, sampled particles, and Gaussians are aligned in one canonical frame, and each Gaussian is tied to neighboring particles so simulated motion can move and rotate it.

Four layers, each building on the last. L0 handles geometry and alignment. L1 turns the reconstruction into a physical object: particles get positions and volumes in metric units, masses are set as m_i = ρV_i from a uniform density prior proposed from the input description, and gravity, ground contact, and zero initial velocities are set. Placement, drop, and slide tests temporarily treat the object as rigid to validate the physical model before deformable simulation begins. L2 proposes Young's modulus, Poisson's ratio, and damping under a homogeneous isotropic neo-Hookean assumption and simulates them with MPM. L3 hands the accepted asset to a robot that plans approach, closure, lift, transport, release, and retreat, and uses the execution to diagnose problems.

Probing instead of guessing. Simulation probes supply evidence for choosing among L2 candidates: deformable drop tests assess impact deformation, rebound, and energy dissipation; compression or lifting probes assess deformation under controlled loading and response after release; a separate longer rollout checks stability. Candidates must be numerically and physically valid, and are ranked by agreement with the requested behavior under fixed test conditions.

A contract-bound loop of agents and a simulator. A Planner selects a permitted action or revision route, a Designer proposes a structured candidate, a Prober evaluates it through geometric checks or simulation, a Critic interprets the evidence to recommend acceptance or revision, and an Orchestrator applies rule checks. The Planner, Designer, and Critic are LLM agents; the Prober runs geometry or simulation; the Orchestrator enforces the harness contract — permitted actions, required observations, hard gates, editable fields, and a revision budget. Only accepted candidates replace the current version, and the search stops when the revision budget is exhausted. Every attempt records its observations, proposal source, and revision history.

Keeping gripper contact physical. The gripper is a kinematic mesh boundary. At MPM grid nodes near the gripper, grid velocity is corrected when the relative normal component points into the gripper, with a friction term retaining a fraction of the tangential relative velocity (Eqs. 1–2). Residual particle penetration and inward relative velocity are corrected after the grid-to-particle transfer. This lets friction support the object without attaching it to the gripper. Mean reaction force on each finger is estimated by summing that finger's contact impulses over a reporting interval and dividing by the interval duration (Eq. 3).

Revision types. Action revisions adjust grasp selection, closure, and motion timing. Numerical revisions tune simulation hyperparameters while preserving action duration. Material revisions address undesired deformation after action diagnosis and require L2 revalidation followed by robot confirmation under the same commanded plan, physical model, and force budget. A per-finger force budget is enforced throughout.

Concrete tooling. Qwen-Image-2512 generates reference images from text; SAM 3 does object segmentation; SAM 3D reconstructs mesh and Gaussians; MoGe-2 supplies camera and metric geometry; CoACD decomposes meshes into convex collision components. Deformable probes and robot interaction use a Warp MPM backend integrated from DeformMaster, with particle-driven Gaussian deformation inspired by SC-GS. Robot simulation uses SAPIEN with a provided URDF of the RealMan RM65-6F robot, and the robot scene is reconstructed with 3D Gaussian Splatting. GraspNet generates grasp poses from point clouds. The harness uses GPT-5.6 Sol by default for the Planner, Designer, and Critic with role-specific prompts and predefined JSON schemas.

Evaluation setup. 39 cases were evaluated — 30 text-driven and 9 image-based, the latter drawn from PhysGen3D's public project assets. GPT-6 Astra (a more capable model than the GPT-5.6 Sol used in the harness) rated physical realism, photorealism, and semantic consistency on a 0–1 scale with the input image and task description as references, following PhysGen3D; these were complemented by blinded pairwise human comparisons following SceneSmith's preference protocol and adapting PhysGen3D's perceptual criteria.

Why This Matters

Research impact. The paper reframes deformable asset generation as a coupled, staged construction problem with simulation in the loop, rather than a one-shot prediction of physical properties from static inputs. This connects agentic generation work (SceneSmith, Scientific Generative Agent) with physics-informed reconstruction (PhysGen3D, PhysGM, SOPHY, PhysX-3D) and motion-based identification (PhysTwin, DeformMaster, PAC-NeRF, Scalable Real2Sim), positioning simulation probes and robot manipulation as the arbiter of whether an inferred asset is actually usable.

Real-world applications:

• Simulation and policy training for deformable objects. DeformSmith produces assets plus replayable manipulation data (commands, particle states, contact observations, outcomes), which can feed policy learning and evaluation for tasks involving soft or squishy objects.
• Content and scene authoring. Generating deformable props from a text prompt or a single photo lowers the cost of populating interactive virtual environments for visualization, games, and embodied AI benchmarks.
• Robotics dataset augmentation. Simulated pick-and-place trials on generated deformable objects can add interaction data that is expensive to collect physically, while failed attempts are retained as labeled diagnostic evidence.
• Initialization for real-object modeling. The paper sketches combining DeformSmith outputs with observation-driven methods such as DeformMaster and EMPM: geometry and physical parameters seed the model, and 3D point tracks from RGB-D video constrain refinement — for example, adjusting Young's modulus to reduce discrepancies between simulated deformation and observed motion.

Industry relevance. Companies building manipulation software, warehouse or logistics robotics touching deformable goods, and teams producing synthetic training data have a direct stake: a pipeline that generates graspable, physically plausible deformable assets with contact-aware validation shortens the loop between asset creation and policy testing. The reliance on named off-the-shelf components (SAM 3, SAM 3D, MoGe-2, CoACD, Warp, SAPIEN, GraspNet, 3D Gaussian Splatting, GPT-5.6 Sol) makes the stack relatively reproducible for well-resourced teams.

Future Directions

• Video-based material identification. The current system infers parameters from text or a single image; the authors motivate future work on video-based identification to validate and correct those estimates against real observations.
• Broader material and contact models. The formulation assumes homogeneous volumetric materials and approximate robot contact, which limits coverage of complex deformable objects. Extending to heterogeneous materials and richer contact models is left open. Fluids and granular materials are explicitly outside the current scope.
• Real-world validation of inferred physical parameters. GPT-6 Astra ratings assess perceived quality under one tested interaction (a five-second drop from one quarter of object height at 30 fps); they do not establish parameter accuracy. Closing that gap, including real robot experiments, remains open.
• Larger and more varied evaluation. The reported study covers 39 cases (30 text, 9 image), three ablations, and six assets with five held-out conditions for the manipulation study — natural next steps include more objects, more interaction tasks beyond pick-and-place, and cross-checking against measured ground-truth material properties.

Target Audience

Researchers and engineers working on physics-based simulation, deformable object manipulation, and generative asset pipelines will gain the most, especially those familiar with MPM, Gaussian-splatting-based rendering, and LLM-agent architectures. Roboticists who need graspable soft-object assets and labeled interaction data, and practitioners building simulators or synthetic data for embodied AI, are the secondary audience. The paper is not beginner-friendly: it presumes comfort with continuum mechanics terminology, simulation backends, and agent-orchestration design, and its quantitative claims are best understood by readers who can evaluate what an LLM-rated 0–1 quality score does and does not measure.

Authors’ abstract

Creating deformable assets for robot manipulation requires jointly specifying their geometry, appearance, and physical properties. This is especially challenging for deformable objects, since text and images provide limited evidence about how they deform and respond to contact, yet these responses directly affect their suitability for interaction. Automated generation therefore needs to resolve coupled physical requirements and use interaction evidence to guide construction and refinement. We present DeformSmith, a framework that enables automated generation of interactive, physically credible deformable assets from text or a single image. Through hierarchical agentic construction and a shared physics-grounded harness, it progressively builds, tests, and refines geometry, physical models, material behavior, and robot interaction until the resulting asset is ready for simulation and manipulation. Robot interaction closes the generation loop through manipulation feedback and replayable interaction data. Results show that DeformSmith generates assets with better visual quality and physical plausibility than state-of-the-art baselines, including PhysGen3D, PhysGM, and PhysX-Omni, while supporting the synthesis of data for robotic manipulation of deformable objects. Project page: https://can-lee.github.io/deformsmith-web/