ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling

AI summary

Overview

Research area: Computer vision and code-generating AI agents; specifically, benchmarks for code-based 3D world construction in Blender.

Technical level: Advanced. The paper assumes familiarity with coding agents, 3D physics simulation, and LLM/VLM evaluation protocols, though the core argument is stated in accessible terms.

Scope (one sentence): The paper introduces ExeWorldBench, a benchmark of three progressively harder tasks that test whether frontier models can turn an understanding of how a mechanical world works into runnable code that actually simulates that world.

What This Paper Is About

Coding agents can already turn an image or text prompt into code that reproduces what a scene looks like, but reproducing appearance is not the same as building a world that behaves correctly. The authors argue that genuine "executable world modeling" needs two linked abilities: world understanding (inferring the structural, mechanical, and causal rules behind a scene) and executable realization (turning that inference into a code-based system whose behavior emerges from physical interactions).

The paper's goal is to measure both abilities together, because existing benchmarks test physical reasoning, visual fidelity, or simple behavior in isolation. ExeWorldBench is presented as the first benchmark to evaluate world understanding and executable realization jointly through code-based 3D world construction.

Key Contributions

1. A formal framing of executable 3D world modeling as the integration of world understanding and executable realization, requiring models to reason about geometric, mechanical, and causal organization and then translate that into an executable code-based system.

2. ExeWorldBench itself, described as the first benchmark to comprehensively evaluate how models translate understanding of "how a world works" into executable realization, built from three tasks of increasing difficulty: Mechanism Executability, Compositional Executability, and Intent-Driven Executable World Synthesis.

3. Two sub-settings for the hardest task: Component-Constrained Synthesis, which supplies the required physical components but leaves their arrangement and causal connections to the model, and Goal-Constrained Synthesis, which supplies only high-level goals, objects, state and event definitions, initial state, physical rules, and final state.

4. A four-part evaluation protocol (Visual Structure, Mechanical Motion, Executable Causal Mechanism, and Goal Satisfaction) applied to frontier models, revealing a substantial gap between code that runs and worlds that are physically and causally coherent.

Main Findings

• Running code is not working code. Repairing only technical execution errors raised executability sharply for API-based models while the world-modeling metrics improved only modestly. For example, on T1, GLM-5.3-Flash went from 26.00% to 78.00% executability but only from 0.48 to 1.52 on Visual Structure and from 0.24 to 0.84 on Mechanical Motion.

• Harness-based models dominate. On T1, GPT-6 Astra (Codex CLI) reached 100% executability with Visual Structure 7.16 and Mechanical Motion 6.32, and Claude Fable 5.1 (Claude Code) reached 100% with 6.44 and 5.68. The best API-based model after repair (GPT-6 Astra) reached 5.84 and 5.08.

• Structure and motion are asymmetric. Models with strong motion scores also score well on structure, but strong structure does not guarantee motion: Claude Fable 5.1 reached V = 4.04 but only M = 3.24 on T1.

• Single-mechanism skill does not transfer to multi-stage systems. After repair, Doubao-Seed-2.1-Pro ranks fifth on T1 Mechanical Motion but seventh on T2 Executable Causal Mechanism (ECM = 0.01).

• Multi-stage composition is far harder than single mechanisms. On T2, GPT-6 Astra (Codex CLI) scored ECM 5.25 and Claude Fable 5.1 (Claude Code) 4.08, while the best API-based result was GPT-6 Astra at 2.47 after repair.

• Synthesis without visual references is the steepest challenge. On T3-1, after repair, GPT-6 Astra reached ECM 3.34 and Claude Fable 5.1 2.75, while all remaining models scored at most 0.40.

• Goal-constrained synthesis collapses for API-based models. On T3-2, harness-based models reached ECM 5.82 and 5.17 and Goal Satisfaction 90.14% and 75.19%; the best API-based model (GPT-6 Astra) reached ECM 2.55 and Goal Satisfaction 41.07% after repair.

• Initial states are easy, causal follow-through is not. Across API-based models, initial-state requirements were generally satisfied more reliably than intermediate-behavior and final-state requirements, indicating models can instantiate a plausible starting configuration but struggle to carry the causal process to the intended outcome.

• Performance degrades with execution horizon. On T2, average ECM decreases steadily with the number of stages; for Claude Fable 5.1 (Claude Code) and GPT-6 Astra, 13-step performance falls to roughly one-fifth of their short-horizon scores. On T3-1, GPT-6 Astra drops below 1.0 ECM at 8 steps.

• Extra motion frames help only marginally. Adding frames from the reference video improved T1 performance for all models except GPT-6 Astra, with generally modest gains; on T2 most models benefited, while GLM-5.3-Flash and GPT-6 Astra showed marginal decreases.

Methodology in Plain English

The authors build three datasets, each with its own pipeline:

• T1 (Mechanism Executability): Publicly available animations of mechanical principles are filtered for visual quality, mechanical complexity, and whether the full motion cycle can be inferred from a static image. Verification uses a visual quality model, GPT-6 Astra for complexity filtering, a generate–infer–validate–verify loop (GPT-6 Astra extracts evidence from the static image, Opus 5 reconstructs the motion without seeing the reference sequence, GPT-6 Astra cross-checks consistency), and manual review by five human annotators.

• T2 (Compositional Executability): A code library of atomic mechanical components with unambiguous motion is assembled. GPT-6 Astra generates ordered component sequences of 4 to 13 steps, each sequence is manually reviewed for physical coherence, and GPT-6 Astra (Codex CLI) assembles the components into a continuous reference animation. The appendix reports 850 component usages, led by Ramp at 256 (30.12%), Catch Bowl at 100 (11.76%), and Domino Chain at 79 (9.29%), down to Multi-Ball Staircase at 3 (0.35%).

• T3-1 (Component-Constrained Synthesis): Components are randomly sampled from the T2 library, 4 to 12 components forming tasks with 4 to 12 stages; GPT-6 Astra checks feasibility (a coherent ordering with plausible stage-to-stage causal links, and every component participating in one execution chain).

• T3-2 (Goal-Constrained Synthesis): GPT-6 Astra generates full task specifications at easy, medium, and hard difficulty; Opus 5 validates internal consistency, mechanical feasibility, and whether a valid reference solution exists; retained tasks get manual review.

Evaluation uses Opus 5.5 as a VLM judge on four metrics: Visual Structure (0–10, static quality and physical validity), Mechanical Motion (0–10, used in T1 for individual component dynamics), Executable Causal Mechanism (0–10 per causal segment, averaged, used in T2 and T3), and Goal Satisfaction (binary per condition on initial state, intermediate behavior, and final state, aggregated by the formula in the paper). Penalized metrics over all examples are reported by default, with conditioned metrics over only code-executable cases in the appendix. API-based outputs are repaired by GPT-6 Astra (Codex CLI) to fix only technical errors without altering scene structure or behavior.

Why This Matters

Impact on research: The benchmark separates two things the field often conflates — a model producing valid code, and a model producing a physically coherent world. Its results suggest that gains in executability are substantially larger than gains in structural, mechanical, or causal correctness, giving the community a concrete target for evaluating agent scaffolds and code-generation methods.

Real-world applications:

• Robotics and embodied-agent training: Generated executable 3D mechanisms could serve as interactive simulation environments, but only if the simulated contacts and causal chains are faithful.
• Game and VR content creation: Procedural generation of working mechanical scenes, not just static geometry.
• Engineering and STEM education: Turning diagrams of mechanisms into runnable, inspectable simulations for teaching how machines work.
• Mechanical design prototyping: Rapid translation of a pictured assembly into a simulatable model that can be tested for motion.

Industry relevance: The gap between harness-based and API-based results suggests that agent scaffolding matters as much as the underlying model for this kind of task, which is directly relevant to teams building coding agents, simulation tooling, and 3D content pipelines.

Future Directions

• Long-horizon causal propagation: The steady decline with stage count, and the drop below 1.0 ECM at 8 steps for GPT-6 Astra on T3-1, points to maintaining valid causal links across many stages as an open problem.

• Closing the understanding-to-realization gap: Models often satisfy initial-state requirements while failing behavior and final-state conditions, so methods are needed that keep an inferred mechanism intact through execution.

• Better use of motion evidence: Motion references produced only modest gains, suggesting models still cannot effectively interpret motion cues and convert them into executable mechanical structure.

• Agent scaffolds and repair strategies: The large advantage of harness-based models over API-based ones, and the fact that repair raised executability far more than world-modeling quality, raises the question of what scaffolding beyond error correction would help.

Target Audience

Researchers and practitioners working on coding agents, LLM-based 3D scene and simulation generation, physical reasoning benchmarks, and embodied AI environment construction. It is also useful for engineering-education and simulation-tooling developers who want to know how far current models are from turning a picture of a mechanism into a working simulation, and for benchmark designers interested in how to evaluate coupled understanding-and-generation capabilities rather than either one alone.

Authors’ abstract

Coding agents are increasingly evolving beyond reproducing what a world looks like toward constructing executable worlds that capture how a world works. This shift calls for executable world modeling, which integrates two core capabilities: world understanding, reasoning about the underlying structural, mechanical, and causal principles of a world, and executable realization, translating that understanding into a code-based system whose behavior emerges from physical interactions. However, existing benchmarks provide only fragmented evaluations centered on physical reasoning, visual fidelity, or behavioral correctness, leaving the integration of these capabilities insufficiently assessed. To bridge this gap, we introduce ExeWorldBench, to our knowledge the first benchmark to comprehensively evaluate world understanding and executable realization through code-based 3D world construction. ExeWorldBench comprises three progressively challenging tasks: Mechanism Executability, which evaluates the realization of individual mechanisms from visual observations; Compositional Executability, which extends to multi-stage systems with spatial, mechanical, and causal dependencies; and Intent-Driven Executable World Synthesis, which requires constructing executable worlds from high-level goals and physical requirements. Extensive experiments on frontier models reveal substantial limitations in executable world modeling, offering insights into the development of more physically grounded and causally coherent world construction systems.