USDCraft: Geometrically Grounded Programmatic Modeling of Articulated 3D Assets for Simulation

AI summary

Overview

Research area: Robotics — specifically articulated 3D asset reconstruction and generation for real-to-sim-to-real robot manipulation, combining LLM-based program synthesis with geometric measurement.

Technical level: Advanced. The paper assumes familiarity with articulated asset representations, joint parameterizations, simulation engines (Isaac Sim/PhysX), benchmark metrics (gIoU, mIoU, Chamfer distance, angular error), and diffusion-policy robot learning.

Scope: The paper introduces USDCraft, a framework in which a pretrained LLM writes, compiles, and iteratively revises executable programs for simulation-ready articulated USD assets grounded in measured source geometry, evaluated on reconstruction, generation, and real-to-sim-to-real manipulation. Note: the provided paper content is truncated partway through the evaluation protocols in the appendix.

What This Paper Is About

Real-world objects often arrive as meshes that are incomplete, corrupted, or fused, and existing mesh-based methods that learn to infer articulation from annotated assets struggle when objects fall outside their training distribution or when the mesh itself is damaged. The goal is to make an LLM build articulated, simulation-ready assets whose dimensions and part placement are anchored to the actual measured geometry of a specific physical object, rather than invented from language priors alone. USDCraft formulates this as programmatic modeling grounded in partial geometric evidence: parts the source mesh observes are corrected against it, while unobserved regions are left open for completion from an image and the object's function.

Key Contributions

1. A formulation of articulated asset reconstruction as programmatic modeling grounded in partial geometric evidence, solved by pretrained LLMs without task-specific training.
2. Source geometry analysis and iterative geometric rechecking — the first converts a source mesh into a metric textual description that distinguishes observed surface from unknown space; the second re-encodes each candidate in the same representation so discrepancies point to specific program edits.
3. Empirical validation of reconstruction accuracy and consistency across diverse assets (USDCraft-bench and Lightwheel), including harness ablations and transfer across model families.
4. Downstream demonstrations of real-to-sim-to-real robot manipulation, plus USDCraft-10k, a library of articulated assets generated with the same toolkit.

Main Findings

• Leading results on USDCraft-bench: Both USDCraft–Sol (GPT-5.6 Sol, high reasoning effort) and USDCraft–Astra (GPT-6 Astra, low reasoning effort) outperform ArtLLM, SIMART, Particulate, and Articraft–Astra on every metric in Table 1. USDCraft–Astra reaches F1 84.830%, static gIoU 0.694, static PC 0.052, mIoU 0.784, articulated gIoU 0.670, articulated PC 0.093, OC 0.010, joint-axis angular error 6.833 degrees, and location error 0.044. For comparison, Articraft–Astra scores F1 64.041, AE 15.106 degrees, LE 0.092; SIMART F1 35.957; Particulate F1 33.366; and ArtLLM (successful outputs only) F1 19.337. USDCraft–Astra results are means over three runs whose variation is far smaller than these margins.

• Competitive results on Lightwheel: On the 243 human-modeled articulated objects in Lightwheel, USDCraft–Astra achieves F1 86.983%, static gIoU 0.590, PC 0.050, mIoU 0.683, articulated gIoU 0.555, PC 0.088, OC 0.011, AE 10.151 degrees, and LE 0.009. With shared plain-text object-function guidance (+G), it reaches F1 89.765%, gIoU 0.631, PC 0.047, mIoU 0.708, articulated gIoU 0.595, and AE 4.463 degrees. Mesh-segmentation methods keep a lower whole-object Chamfer distance, and Instruct-Particulate yields a higher mIoU (0.724 on dataset meshes vs. 0.708 for USDCraft–Astra +G), because segmentation reproduces the supplied surface exactly while USDCraft rebuilds every part and accumulates small surface deviations.

• Joint errors need context: Articraft–Astra shows a lower angular error on Lightwheel (7.657 degrees), but joint errors are averaged only over matched parts, and Articraft matches only about two-thirds of the reference movable parts (recall 66.8% vs. 85.0% for USDCraft).

• The harness matters more than the backbone: Compared against a Mini Workflow that uses the same Sol or Astra backbone with general-purpose tools, USDCraft improves all nine reconstruction metrics with both backbones. USDCraft–Sol outperforms Mini Workflow–Astra on every metric (F1 79.835 vs. 74.480; AE 9.155 vs. 21.141 degrees). USDCraft–Claude (Claude Opus 5) and USDCraft–Gemini (Gemini 3.8 Flash) also beat Mini Workflow on nearly all metrics, showing the harness transfers across model families without modification. The added analysis and rechecking cost about one to two extra minutes per asset for the GPT backbones.

• Sim-to-real drop is the informative quantity: In the real-to-sim-to-real study, USDCraft loses at most 10 points on any task, whereas Articraft, built from the image alone, loses 60–80 points, and Mini Workflow varies widely. USDCraft–Astra achieves 100% simulation / 90% real on open drawer, 90% / 85% on press toast, and 85% / 85% on turn on toaster. Particulate's incomplete reconstructions support only lever pressing (50% sim / 10% real), indicating structural completeness must come before geometric accuracy.

• Cumulative ablations: The authoring harness gives the largest single gain (F1 74.480 to 80.925; AE 21.141 to 9.201 degrees). Source geometry analysis brings the next largest gain (F1 84.030; AE 7.337 degrees) and removing it causes the largest drop in part recovery, geometric overlap, and joint accuracy. Iterative rechecking mainly improves geometric fidelity and joint placement. Removing visual feedback leaves geometry essentially unchanged but lowers part recovery and joint accuracy. Adding an independent observation agent lowers every metric.

• Generation quality and speed: On 50 text-conditioned and 50 image-conditioned requests per setting, USDCraft scores highest in every dimension under an anonymous Sol/high evaluator. USDCraft–Astra takes 5.6 minutes mean / 4.9 minutes median, versus 12.0 / 10.3 for Articraft–Sol and 58.8 / 58.7 for Procedura–Sol. The largest gaps lie in condition adherence and appearance.

• A generated asset library: USDCraft-10k contains 10,000 assets spanning more than 500 everyday object categories, produced entirely with USDCraft–Astra from 70% text-conditioned and 30% image-conditioned requests.

• Benchmark composition: USDCraft-bench contains 60 agent-generated assets (20 each from USDCraft, Articraft, and Procedura) plus a 40-object extension combining meshes from Hunyuan3D 2.1 and real scans. Under the paper's functional grouping it covers 48 categories; Lightwheel covers 14; seven overlap, giving 55 distinct categories in total.

• Grounding in the reference frame: When every prediction is evaluated in the reference frame (retaining absolute size and position errors), USDCraft's scores change little and remain best on all nine metrics, whereas image-only Articraft drops sharply because its output lacks a metric scale.

Methodology in Plain English

USDCraft treats building an articulated asset as writing a program, not

Authors’ abstract

Geometrically faithful and functional articulated 3D assets are essential for real-to-sim robot manipulation, where policies trained in simulation must transfer to physical objects. Recent mesh-based methods learn to infer articulation from annotated 3D assets, but deployment remains challenging when real-world objects fall outside the training distribution or their meshes are incomplete or corrupted. To address these limitations, we formulate articulated asset reconstruction as programmatic modeling grounded in partial geometric evidence and introduce USDCraft, a framework in which a pretrained LLM writes and revises executable programs for simulation-ready articulated assets without task-specific training. We propose source geometry analysis, which converts the source mesh into a metric textual description that distinguishes observed surface from unknown space, and iterative geometric rechecking, which re-encodes each candidate in the same representation so that discrepancies point to program edits while unobserved regions remain open to completion. Visual feedback and physical authoring guidance complete the modeling process, which produces articulated USD assets with explicit physical properties that load into Isaac Sim without manual adjustment. Experiments demonstrate leading articulation recovery on two benchmarks and validate USDCraft's effectiveness for real-to-sim-to-real robot manipulation.