Towards Physically Grounded Articulated 3D Assets: A Multi-Agent Pipeline and Benchmark

AI summary

Overview

• Research area: Computer vision and 3D content generation, specifically text- and image-conditioned generation of articulated 3D objects with physical properties, evaluated with a new benchmark spanning spatial fidelity, physical plausibility, downstream robotic task performance, and human judgment.
• Technical level: Intermediate. Understanding the paper's details requires some familiarity with 3D reconstruction metrics, physics simulators, and LLM-based agent pipelines, but the core idea is accessible.
• One-sentence scope: The paper introduces Artisan, a three-agent LLM pipeline with physics-simulator feedback that generates articulated 3D assets with collision-free motion, gravity stability, mass and material properties, and pairs it with a unified benchmark covering 100 reference assets from ArtVIP and LightWheel, 50 stability prompts, and 50 ABO objects with reference masses, evaluated across more than 10 representative models.

What This Paper Is About

Most 3D generation methods treat shape, appearance, articulation, mass, material, stability, and collision-free motion as separate problems, which produces assets that look plausible but fail when placed in a physics simulator. This paper asks how to generate articulated 3D assets from a natural-language prompt and optional reference image such that the result is not only visually faithful but physically valid: parts do not interpenetrate during motion, freestanding objects stay upright under gravity, and the object can be grasped and manipulated. The authors build a system called Artisan for this task and a benchmark to measure whether the goal is actually met.

Key Contributions

1. Artisan, a multi-agent generation pipeline. Building on the blueprint of Articraft (Zhou et al., 2026), the system splits generation across three independent LLM agents — a Design Agent that produces a structured textual blueprint and design board, a Coding Agent that translates the blueprint into an executable Python program producing geometry, articulations, and physical parameters, and a Review Agent that scores the asset and returns a structured issue list. The agents exchange only the blueprint, the 3D asset, and feedback, which the authors argue reduces self-evaluation bias where a single agent satisfies its own checks.

2. Physical simulator feedback inside the generation loop. Candidate assets are validated in MuJoCo with collision checks (convex decomposition followed by simulation of single-joint, double-joint, and multi-joint motion) and stability checks (simulation under gravity for up to 5 seconds, with tilt recorded). The resulting physics report guides coding repairs and review, so critical structural or functional failures cannot be offset by a high overall score.

3. A unified benchmark for physically grounded articulated assets. The benchmark spans four aspects — spatial fidelity, physical plausibility, downstream task performance, and overall quality — with multiple metrics per aspect, over 100 reference assets from ArtVIP and LightWheel, 50 stability prompts, and 50 ABO objects with reference masses.

4. An extension of the Articraft SDK with reusable standard connectors (six families of parameterized parts such as hinges, slide rails, and rotary connectors) and automatic action annotation, where the Coding Agent records the target rigid part, grasp/press region, joint to actuate, and required fraction of joint travel into a manifest beside the exported meshes.

Main Findings

• Best physical plausibility among evaluated methods. Artisan (GPT-6 Astra) reduces collisions to 1.7% in single-joint, 3.7% in double-joint, and 13.0% in multi-joint evaluations on ArtVIP, and reaches 98% stability in the rest state and 96% in the articulated state — described as the lowest collision rates and highest stability rates among the evaluated methods. By comparison, Articraft (GPT-6 Astra) reports 45.5%, 60.7%, and 78.8% collisions with 82% and 82% stability.

• Competitive spatial fidelity. Artisan (GPT-6 Astra) achieves IoU 0.7675, SSIM 0.8769, PSNR 18.65, LPIPS 0.2319, and CLIP 0.9239, with rest-pose oCD 0.0011, pCD 0.0101, pIoU 0.3587 and articulated oCD 0.0021, pCD 0.0160, pIoU 0.3417 on ArtVIP. The authors state it outperforms other methods in 8 of 11 spatial-fidelity metrics in Table 1.

• Physical validation is the decisive ingredient. Ablating the physical validation (Ours GPT-5.5 w/o phys.) leaves appearance, geometry, and even kinematics comparable to the full system, but collision rates rise sharply (37.5% single, 47.3% double, 77.6% multi), indicating those assets cannot execute their intended motion in the simulator. In Figure 5, Artisan is described as the only method that avoids component intersections and stays upright while five baselines topple.

• Multi-agent design and independent review eliminate self-evaluation bias. Comparing "Ours (GPT-5.5 w/o phys.)" with "Articraft (GPT-5.5)" yields average improvements of 5.4% in pCD, 4.9% in pIoU, and 11.9% in collision rates.

• Agentic methods rival or surpass task-specific models. Against Instruct-Particulate (whose input mesh is generated by Hunyuan-3.1 from the reference image), Artisan achieves superior results in most metrics in Tables 1 and 2.

• Better downstream robotic utility. Fine-tuning the vision-language-action policy π0.5 on Artisan assets reaches 63% success on RoboTwin 2.0 and 27% on RoboCasa365, versus 60% and 5% for Articraft assets, and 80% and 39% for native-only training. Mixing native and Artisan assets gives the best result: 84% on RoboTwin and 43% on RoboCasa365.

• Higher asset qualification rates. Artisan assets pass Round 1 (physical stability and task feasibility) at 100% on both benchmarks and Round 2 (demonstration generation success) at 100% (RoboTwin) and 44% (RoboCasa365), compared with Articraft at 62%/62% and 39%/8%. The paper reports the average advantage over Articraft as 37 percentage points across the two rounds.

• Winning human preference. In human evaluation with six anonymized methods, Artisan received 43.3% of Top-1 votes and 85.1% of Top-3 votes, ahead of the runner-up Codex at 21.1% Top-1 and 68.7% Top-3.

• Mass estimation is mixed. On the 50 ABO objects with reference masses, Artisan (GPT-6 Astra) reports MnRE 0.773, ADE 5.86 kg, and APE 0.260; the best reported ADE among the listed methods is Agy (Gemini-Flash-3.8) at 5.49 kg, and the best APE is Artisan (GPT-6 Astra) at 0.260, while PUGS reports APE 3.016.

Methodology in Plain English

The system starts from a natural-language prompt and optional reference images. A Design Agent converts them into a textual blueprint: what parts the object has, what materials they use, how parts are parented to each other, what joints connect them, and what functional behaviors they should support (for example, a bin lid opening when its pedal is pressed, or a mixing head lifting out of a bowl). The same agent also produces a design board — generated images showing orthographic views, an exploded view of the parts, a kinematic diagram in closed and open states, and cutaway sections of interior parts.

A Coding Agent then writes a Python program that builds the asset, using the Articraft SDK plus newly added reusable standard connectors such as hinges, slide rails, and rotary connectors, so the agent only picks a connector and adjusts its dimensions and mounting positions rather than designing a joint from scratch. The program is executed and the resulting asset is inspected using geometric and visual tools. For each interaction task named in the blueprint, the agent records which part is grasped or pressed, which joint is actuated, and how much joint travel success requires.

A Review Agent independently examines the asset, the blueprint, the design board, and the physics report, producing scores per evaluation criterion and a structured list of issues, each with the affected component, supporting evidence, severity, and what needs to be fixed. The physical report comes from MuJoCo: parts are checked for interpenetration across sampled single-joint, double-joint, and multi-joint configurations, and the assembly is simulated under gravity to measure tilt. The review feedback drives the next coding iteration, but the Review Agent never edits the program itself. The loop repeats until the criteria are satisfied.

For the benchmark, the authors combine standard 3D reconstruction metrics with physical metrics. Appearance is compared by rendering both generated and reference assets in Blender at 1024² resolution from three viewpoints per state and computing IoU, PSNR, SSIM, LPIPS, and CLIP. Geometry uses Chamfer distance at object and part level plus part IoU. Materials are compared by rendering colour, metallic, and roughness buffers and computing PSNR, with a protocol to transfer predicted materials onto ground-truth geometry. Mass uses absolute difference in kilograms, absolute percentage error, and a min-ratio error. Collision is measured over 9 uniformly sampled states for single joints, a 9×9 grid for two joints, and 200 random trials for multi-joint motion, excluding pairs that also collide on the ground-truth asset. Stability simulates 50 objects with high centres of mass for 10 seconds in rest and fully articulated states, counting an object stable only if its maximum tilt stays below 30°, its positional change below 1 mm, and its rotational change at most 0.01 rad. Finally, assets are tested as training data for the π0.5 robotic policy and ranked by human participants who manipulate them in MuJoCo, with six anonymized method outputs presented in random order.

Why This Matters

Research impact. The paper argues that visual fidelity alone does not tell you whether a generated asset will function, and supplies both an ablation showing physical validation is what makes the difference and a benchmark that measures collision, stability, mass, material, downstream task success, and human preference together. It also frames agentic harnessing itself as an emerging research area, noting that a general-purpose pipeline can beat task-specific feed-forward reconstruction models in most metrics, which invites re-examination of how feed-forward models are used.

Real-world applications named or implied by the paper:

• Generating simulation-ready assets for physics engines, reducing the cost of building environments for simulation.
• Product design workflows, where generated articulated objects can be inspected and edited.
• Robotics, where assets populate training data for manipulation policies — the paper demonstrates this with π0.5 fine-tuned on Artisan assets and with the RoboTwin 2.0 and RoboCasa365 evaluations.
• Virtual laboratory and training environments, referenced in the acknowledgements as a concept suggested during the work.

Industry relevance. Training data for robot learning is expensive to collect, and native asset pools are limited; the paper shows that generated assets can supplement native training sets and raise success rates on held-out instances (84% on RoboTwin, 43% on RoboCasa365 when mixing native and Artisan assets). Asset qualification rates of 100% in Round 1 for Artisan also speak directly to whether generated content is usable without manual repair.

Future Directions

• Revisiting the role of feed-forward models in light of agentic pipelines that outperform them on most benchmark metrics.
• Developing efficient agent harnesses that generalize across tasks, rather than tuning a pipeline per task.
• Providing richer and more actionable feedback from simulators so agents can repair physical defects more precisely.
• Extending simulators to capture a broader range of physical phenomena than the collision and gravity stability tested here.
• The paper also notes limitations of the current evaluation — for instance, the human study reports aggregate preference statistics, and the number of participants is not reported in the available content.

Target Audience

Researchers and practitioners working on 3D asset generation, articulated object modelling, physics-based simulation content, and robot learning data generation. It is most useful for readers who care about whether generated assets function in a simulator, want a benchmark with physical plausibility and downstream robotic metrics rather than appearance alone, or are designing LLM-based agent pipelines with tool and simulator feedback. Readers primarily interested in pure visual quality metrics will find the physical and robotic evaluation sections the most novel part.

Authors’ abstract

In this paper, we study physically grounded articulated 3D asset generation---moving the goal from reproducing how objects look to capturing how they move and function. Inspired by the literature on procedural and agentic generation, we present Artisan, a system that extends the latest method Articraft by introducing a multi-agent pipeline with physical simulation feedback, enabling comprehensive attributes beyond geometry, appearance, and physically plausible articulation, such as collision-free motion, stability under gravity, and accurate mass and material properties. Furthermore, we construct a unified benchmark spanning four key aspects: spatial fidelity, physical plausibility, downstream task performance, and overall quality. The benchmark encompasses 100 reference assets from ArtVIP and LightWheel, 50 stability prompts, and 50 ABO objects with reference masses, with protocols evaluating various metrics. Through extensive evaluation of more than 10 representative models, Artisan achieves competitive spatial fidelity while attaining the lowest collision rates and highest stability rates among the evaluated methods. Together, our system and benchmark establish insights for advancing 3D generation towards physically plausible, functionally usable assets.