3DCodeVerse: Exploring 3D Worlds as Code

AI summary

Overview

Research area: Computer vision and 3D content generation, specifically generating 3D assets through executable code (Blender Python, CadQuery, Three.js, OpenSCAD, GLSL/GLSL-style shaders, URDF) driven by large language models and coding agents.

Technical level: Advanced. The paper assumes familiarity with LLM fine-tuning (LoRA, SFT, DPO), multi-agent systems, and 3D representations such as meshes, URDF articulation, and scene graphs.

Scope (one sentence): The paper introduces 3DCodeVerse, an open ecosystem consisting of a 2.03M-pair dataset of executable 3D programs, a multi-agent harness that generates and automatically verifies 3D code, and baseline models with training recipes.

What This Paper Is About

Producing 3D content as code (rather than as a mesh or a NeRF) yields structured, editable, reusable artifacts, but the field lacks shared infrastructure: existing datasets focus narrowly on static objects or CAD parts, and existing agent systems each support only one environment and one style of feedback. The authors build a unified corpus of standalone executable 3D programs, a harness that plans, writes, executes, measures, judges, and repairs 3D programs without human feedback during a run, and open baselines that show the data and harness both measurably improve 3D coding.

Key Contributions

1. 3DCodeVerse-2M, a curated dataset of 2.03M pairs of executable 3D programs spanning six code types across static objects, articulated objects, scenes, and graphics. Each record bundles source files, required assets, three captions (instruction, visual description, construction procedure), provenance, license, and available verification records. Unlike prior corpora, the programs are made standalone by translating specialized APIs and inlining routines (for example, converting DeepCAD construction sequences into CadQuery programs and inlining Articraft geometry while preserving URDF articulation).

2. 3DCodeVerse Harness, a multi-agent directed acyclic graph framework that decomposes a prompt into a structural plan with named parts, metric bounding boxes, materials, and an acceptance checklist, dispatches file-scoped coding sub-agents, assembles their outputs, and then runs repeated execution–measurement–judgment–repair cycles. Feedback comes from three sources: build/runtime logs, deterministic spatial tools (dimension, contact, collision, kinematics, placement checks), and a vision–language judge inspecting multi-view renders.

3. A dual-use data engine. The same harness that acts as an inference system for content creation also records successful programs, failed attempts, targeted repairs, and preference comparisons, which supply supervision for supervised fine-tuning and preference learning.

4. Systematic empirical study and open baselines, including fine-tuned 9B and 27B open models, per-dialect fidelity measurements, pairwise human and VLM evaluations against native harnesses and Claude Design, and an "adversarial loop engineering" procedure that alternates between refining the VLM judge and refining the generation workflow.

Main Findings

• Fine-tuning makes open models executable. Qwen3.5-9B goes from 0.0% execution on 3DCodeBench to 84.9% after LoRA SFT (+84.9 points); it rises on Blender from 1.9% to 90.3% (+88.4), on CadQuery from 10.5% to 98.5% (+88.0), on OpenSCAD from 50.0% to 86.0% (+36.0), on GLSL from 33.0% to 56.0% (+23.0), and on Three.js from 72.5% to 74.5% (+2.0).

• The larger model benefits too. Qwen3.8-27B with SFT + DPO goes from 51.9% to 92.9% on 3DCodeBench (+41.0), to 93.2% on Blender (+18.4), 98.5% on CadQuery (+33.0), 90.0% on OpenSCAD (+10.0), 71.5% on GLSL (+14.5), and 92.5% on Three.js (+20.0).

• Failure before training was mostly dialect-level. On 3DCodeBench, none of Qwen3.5-9B's programs executed before fine-tuning, most failing immediately by calling scene-clearing operators unavailable in Blender 5.

• The harness beats native harnesses. On 3D objects, Human Preference for the 3DCodeVerse Harness ranges from 69% to 89%, with Judge Score improving by 11–25 points. The largest gains are for weaker models: Gemini 3.8 Flash moves from 61 to 87 and GPT-6 Luna from 64 to 86.

• Strong result against a commercial design system. Against Claude Design (3D Object), with Claude Opus 5.5 driving both systems, the 3DCodeVerse Harness is preferred in 90% of human pairwise comparisons and scores 88 versus 37 on Judge Score.

• Gains generalize beyond objects. Articulated objects, scenes, and graphics reach 78–88% Human Preference, and the harness yields 15.5–29.1 point gains over one-shot generation.

• Human and rubric judgments agree. The authors note that agreement between human preference and rubric-based scores suggests the gains are not an artifact of the VLM rubric used during refinement.

• Fidelity to reference improves with fine-tuning. Qwen3.5-9B's F@0.05 goes from .000 to .317 and DINOv2-L similarity from .000 to .266; per-dialect F@0.05 rises from .008 to .693 on Blender and from .030 to .308 on CadQuery. Qwen3.8-27B goes from .214 to .374 on F and .211 to .298 on DINO.

• Preliminary robotics check favors code renders. Across 12 cases spanning two assets, two contact states, and three control sequences, code renders and their UWM VAE reconstructions each pass a fixed door-and-handle geometry check in 12/12 outputs, versus 7/36 for frozen UWM predictions.

• Evaluation scale. Results are measured on 805 fixed tasks across six suites: 212 text-to-Blender tasks from 3DCodeBench, plus Blender (103), CadQuery (200), OpenSCAD (50), GLSL (200), and Three.js (40) splits drawn from 3DCodeVerse-2M.

Methodology in Plain English

The authors define 3D coding as writing a program that, when run in a declared environment such as Blender's bpy or CadQuery, produces a 3D artifact. Success requires both that the program runs (execution) and that the artifact is good (quality), measured either against a reference via a distance metric or against the request via programmatic measurements and renders.

Building the dataset. They combine three routes. First, collection from public sources including Bioinspired3D, 3DCodeBench, Thingiverse, Shadertoy, CodePen animations, and multi-file Three.js / React Three Fiber projects. Second, conversion: DeepCAD construction sequences are translated into CadQuery by a deterministic script, Articraft geometry routines are inlined into CadQuery while retaining the URDF, and a ProcTHOR subset is paired with Blender/Three.js reconstructions. Conversion fidelity is checked with voxel IoU for DeepCAD and Chamfer distance for Articraft. Third, generation: coding agents run the harness on text and image prompts to synthesize new programs. Agents use reusable "skills" that package authoring conventions, code examples, and repair guidance to make each program standalone. A VLM then captions each program from code and renders under three conditions: an instruction, a visual description, and a construction procedure.

Building the harness. A planner turns the request into a structured plan with metric bounding boxes, materials, attachment topology, and an acceptance checklist; for articulated objects the plan is a kinematic tree with typed joints, axes, pivots, and limits. The plan becomes a DAG of coding and assembly tasks with explicit file ownership, so independent sub-agents work concurrently on separate files. After assembly, the program is executed and rendered. Deterministic gates (connectivity, contact ledger, contract checks against planned dimensions within per-axis tolerances) run before any model judgment, and gate failures cap the round's score while gate passes veto contradictory visual claims from the judge. Corrective hints are projected back into each runtime's native authoring frame, so a seat floating by 12 mm yields a specific corrective translation vector in Z-up Blender Python versus Y-up Three.js. A fix agent applies prioritized repair tasks, and the highest-scoring candidate across rounds is selected.

Tuning the feedback loop itself. Rather than trusting the judge out of the box, the authors alternate between two optimization steps. They first freeze the generation workflow, replay a bank of stored artifacts, and revise the judge's view selection, evidence presentation, defect checklists, and score aggregation to reduce judgment variance and contradictions with measured evidence (the judge must state observations before scores and never sees the generator's reasoning). Then, with the judge fixed, they revise the harness's prompts, contracts, skills, tools, and refinement policy on matched prompts, balancing judged quality against observable workflow failures such as invalid plans, retried tool calls, and failed builds.

Evaluation. Human Preference is the fraction of anonymized side-by-side comparisons won, ties split equally; VLM Preference is the same computed by a Gemini 3.1 Pro judge; Judge Score is that judge's 0–100 rubric score with failed generations scored zero. They also report SigLIP-2 text–image cosine similarity averaged over four views, following 3DCodeBench. For fine-tuning, models are trained with LoRA in LLaMA-Factory, decoded at T=0.7 with one seed, and every output is executed in the dialect's pinned runtime.

Why This Matters

Impact on research. The paper reframes 3D generation infrastructure as a closed loop — data construction, program synthesis, verification, refinement, and model improvement — rather than as separate dataset, benchmark, and agent papers. Its key argument is that verification signals (deterministic geometry gates plus blind VLM judgment) are not just evaluation tools but are themselves a scalable source of training supervision, since the harness can generate preference pairs and repair traces without a human in the loop. The released dataset, harness, models, and recipes give other groups a common substrate across Blender, CadQuery, OpenSCAD, Three.js, GLSL, and URDF, where previously each environment had its own isolated corpus.

Real-world applications (as presented or implied by the paper):

• Robotics simulation. Articulated assets with explicit rigid-link geometry, joint axes, and joint limits can be exported to MuJoCo as editable task instances, with programmatic variation of dimensions, articulation, and initial poses.
• Video generation conditioning. Rendered 3D code guides video models: the authors condition Gemini Omni Flash 1.1 for robot manipulation scenes and Seedance 2.5 for a GPT-6 Astra scene, using the renders for spatial and temporal guidance while the video model refines appearance.
• Editable 3D asset authoring. Because output is source code with bundled assets, users can edit, reuse, and compose content and inspect it, rather than receiving an opaque mesh.
• Mechanical and CAD design. CadQuery and OpenSCAD tracks target parametric parts, where dimensions and constraints are explicit and checkable.

Industry relevance. The 90% human win rate against Claude Design (3D Object) with the same backbone model is the paper's headline commercial claim: it suggests that harness engineering — tool design, verification gates, and rubric design — can matter as much as the underlying model for structured 3D output. That is directly actionable for teams building coding-agent products, game and simulation asset pipelines, and any workflow where generated content must pass geometric rather than merely visual checks.

Future Directions

• Closing the gap on browser graphics. Three.js execution only moved from 72.5% to 74.5% for the 9B model and OpenSCAD F@0.05 moved just .001 for the 27B model, so several dialects remain far from saturated.

• Scaling the data engine. The authors describe 3DCodeVerse-2M's construction as supporting "continued data expansion"; how far execution rate and fidelity climb with more harness-generated, verified programs is an open question.

• More reliable automated judgment. The adversarial loop engineering section exists because judge reliability depends on prompt, tool, and rubric configuration; the residual variance and the risk of a rubric that rewards what the judge can see rather than what is correct remain live concerns.

• Broader validation beyond preliminary robotics evidence. The robotics result covers only 12 cases, two assets, two contact states, and three control sequences, and is explicitly framed as a preliminary evaluation, leaving larger-scale physical-plausibility validation open.

• Extending inference-time scaling. The harness supports projects with thousands of lines of code across seven runtimes; whether the same DAG and gate design transfers to additional engines and longer-horizon multi-file projects is untested here.

Target Audience

Researchers and engineers working on 3D generative models, LLM-based coding agents, and agentic tool-use systems will get the most from this paper, particularly those interested in verification-driven generation and in building training data from agent execution traces. It is also useful for practitioners in robotics simulation, CAD/parametric design, and technical art who need editable, checkable 3D output rather than meshes, and for teams evaluating whether harness design can substitute for or complement a stronger base model. Readers without background in 3D representations or LLM fine-tuning will find the appendix-level implementation detail dense, though the main results and the conceptual framing of "3D worlds as code" are accessible.

Authors’ abstract

We introduce 3DCodeVerse, a research ecosystem that enables coding agents to create editable and verifiable 3D content at scale. 3DCodeVerse comprises large-scale, high-quality datasets of executable 3D code, a coding harness for spatial reasoning and automated verification, and practical guidance for 3D coding. First, we construct 3DCodeVerse-2M, a large-scale curated dataset of standalone 3D executable code. We build this corpus by collecting open-source data, converting specialized representations, and generating new programs with coding agents. Each program is packaged with the source files and associated assets required for execution in its declared environment, along with available descriptions and verification records. Second, we develop the 3DCodeVerse Harness, a multi-agent directed acyclic graph framework for generating and refining 3D content. It uses execution checks, spatial measurements, and visual feedback to automatically evaluate and revise generated programs without human feedback during generation. Third, we conduct systematic empirical studies of 3D coding, establishing baselines with fine-tuned open models and deriving practical guidance through component-level analyses of our harness. Extensive experiments demonstrate that fine-tuning on our dataset significantly improves the 3D coding capabilities of open-source models. Notably, our 3DCodeVerse harness achieves a 90% win rate against Claude Design (3D Object) in pairwise human evaluations, when both are utilized by Claude Opus 5.5.