Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning

AI summary

Overview

Research area: Computer vision and multimodal representation learning, at the intersection of physical reasoning, simulation, and vision-language model (VLM) training.

Technical level: Advanced. The paper assumes familiarity with vision-language models, reinforcement learning from verifiable rewards (GRPO), physical simulators, and 3D/depth-based scene reconstruction.

Scope: The paper proposes Code-as-World, a paradigm that represents physical worlds as executable code and uses an agentic propose–instantiate–execute–render–verify loop to discover those representations from text or video, then applies them as supervision for quantitative physical reasoning in VLMs (reported as arXiv:2608.27549v1, MirroS Technical Report, August 27, 2026).

What This Paper Is About

Modern VLMs can describe physical events but do not build explicit representations of the mechanisms behind them—object states, physical parameters, and governing dynamics—so they struggle to reason about how a scene would evolve or respond to interventions. The authors frame this as a "phenomenon–mechanism dichotomy": describing what happens is not the same as recovering why it happens. Their goal is a representation that is compact, quantitatively grounded, editable, and executable, discovered automatically from natural-language descriptions or real videos, and usable as scalable physical supervision for training VLMs.

Key Contributions

1. Executable world representation (EWR). After a discussion of existing physical world representations (pixels, 3D/reconstruction, natural language), the paper introduces Code-as-World, which represents task-relevant physical worlds as executable code decomposed into physical composition (C), dynamic evolution (E), and visual appearance (A).
2. Agentic discovery loop. The paper formulates world representation as an agentic discovery problem rather than one-shot prediction, developing a shared propose–instantiate–execute–render–verify loop that iteratively searches for world representations consistent with language or visual evidence.
3. Physical supervision for VLMs. Verified executable worlds are used to generate scalable supervision for quantitative physical reasoning; the resulting Code-as-World-VL family is reported to achieve state-of-the-art performance on QuantiPhy and to surpass leading proprietary models.

Main Findings

• Executable code as a representation space: An EWR is written as p = (C, E, A), where C specifies objects, geometry, metric dimensions, mass, friction, gravity, and environment structures such as floors, tables, and walls; E specifies initial states, temporal changes, key events, and simulation duration; A specifies camera parameters, backgrounds, materials, lighting, frame rates, resolutions, and rendering configurations. Components can be inspected, modified, and executed independently.
• Discovery loop effectiveness: With a maximum of K = 5 agentic discovery rounds, iterative refinement improved static quality and most motion-fidelity metrics across rounds relative to one-shot generation. At a matched five-evaluation budget, the agentic loop outperformed Best-of-5 independent sampling on Visual Alignment, Object IoU, Traj-ADE, and Accuracy@2%D (Traj-ADE reported in %D, Velocity-ADE in %D/step).
• Controllable resimulation: Recovered worlds behave as editable programs. The paper shows a bowling ball's initial velocity direction being changed to produce distinct trajectories, and camera configurations changed to render the same collision from a global view or from either car, with an internal video generation model rendering edits as realistic video.
• Quantitative physical reasoning on QuantiPhy-validation: Code-as-World-VL-9B scored 55.4 average (2S: 55.0, 2D: 52.9, 3S: 55.6, 3D: 58.1), exceeding Gemini-3.1 Flash (54.8 average; 49.4 / 47.5 / 61.4 / 61.1) and all other evaluated baselines. Code-as-World-VL-27B (Reasoning) scored 58.6 average (48.7 / 62.4 / 60.5 / 62.8). Code-as-World-VL-4B scored 50.6 average (45.4 / 55.4 / 45.8 / 56.0).
• Other baselines for context: Among open-weight models, the highest reported average was Qwen3-VL-32B-Instruct at 40.2, followed by InternVL-3.5-30B at 35.5, Qwen3-VL-8B-Instruct at 32.3, Qwen3.5-4B at 31.2, and InternVL-3.5-8B at 30.0. Among proprietary models, ChatGPT-5.1 averaged 48.4, Gemini-2.5 Pro 46.4, Gemini-2.5 Flash 44.1, Grok 4.1 (Fast Reasoning) 36.8, ChatGPT-5 32.6, and ChatGPT-5 Pro 19.5.
• Reasoning behavior of the 27B variant: The paper shows two abridged traces. In a bicycle example using provided priors (bicycle–camera distances of 23.5534 m at 0.58 s and 23.2262 m at 1.58 s), the model predicted 6 m/s against a ground truth of 5.33 m/s (MRA 0.8). In a billiards example with a 57.2 mm ball diameter prior (from internet_0027), it predicted 26.8 cm/s against 26.82 cm/s (MRA 1.0).
• Image-space grounding is preserved: The complete Code-as-World-VL models improved over their Image-Space counterparts on every benchmark, indicating that world-space training preserves and strengthens image-space grounding. The paper references Table 3 and Appendix B.3 for the complete image-space results, which are not included in the available content.
• Stated limitation: Current simulators cannot faithfully capture the full range of physical conditions, and seemingly simple rigid-body motion can be sensitive to small variations in terrain, contact geometry, material properties, or other latent factors. When a process falls outside the simulator's modeling scope, the discovery loop may converge to a locally plausible EWR without recovering a mechanistically accurate explanation.

Methodology in Plain English

The system represents a physical scene as a short program rather than as pixels, latent features, or a caption. That program has three parts: what the scene contains (objects, sizes, masses, friction, gravity, and static structures), how it changes over time (initial states, events, duration), and how it looks (camera, lighting, materials, resolution).

To recover such a program from evidence, the authors use an agentic loop inspired by abductive reasoning—the scientific move of inferring the mechanism that best explains observations. For text input, an agent extracts entities, spatial relations, events, and intended outcomes as semantic evidence. For video input, it extracts depth maps, instance masks, and object tracks as visual evidence, and uses a 3D object generation model to build meshes for segmented objects, recovering positions, scales, and dynamic states.

The loop then repeats: propose or modify an EWR, compile it into simulator-ready parameters, execute it to produce a full state trajectory with contacts and collisions, render and project it back into the input view (RGB, depth, masks, trajectories), and compare against the evidence at selected key frames. Discrepancies are turned into structured feedback that guides a local revision. The loop stops when the hypothesis explains the evidence sufficiently well and parsimoniously, or rejects the hypothesis when the iteration budget is exhausted. A world is retained only if it can be rendered and verified against the source video.

For the downstream VLM application, the authors train in two phases. Phase one teaches image-space measurement via supervised fine-tuning on pixel-level question–answer pairs built from RefCOCO, RefCOCO+, RefCOCOg, RefCLEF, and GOT-10K (questions about extent, position, displacement, velocity, and acceleration in pixels). Phase two applies Group Relative Policy Optimization on world-level VQA derived from text-driven and video-driven executable worlds, using a reward that combines scale-normalized numerical accuracy with auxiliary rewards for unit correctness and response format. Text-driven worlds give fully observable, numerically exact physical supervision; video-driven worlds better match real video appearance and motion. Training used eight NVIDIA H100 GPUs, with 16 temporally ordered frames sampled per video.

Why This Matters

Impact on research. The paper positions executable code as a representation that is semantic like language, structured like reconstruction, and capable of modeling temporal evolution like generative models—while remaining quantitatively grounded. It also reframes world model construction as a search and verification problem rather than a one-shot prediction problem, and shows that verified worlds can serve as scalable supervision where real-world videos lack annotations of underlying physical quantities.

Real-world applications:

• Generating counterfactual physical videos by editing a recovered world (changing initial velocities, physical quantities, or camera viewpoints) and re-rendering, without reconstructing every variant.
• Sim-to-real content generation: turning text specifications into simulator rollouts and then into realistic videos with richer objects, materials, backgrounds, and textures while preserving the encoded motion.
• Training vision-language models for measurement and metric physical reasoning in videos, including size, displacement, velocity, and acceleration estimation from monocular footage.
• Providing verifiable reward signals and physical state data for downstream reasoning, simulation, and data generation pipelines.

Industry relevance. The reported results target applications where models must reason about how physical scenes respond to change rather than merely describe them—robotics and embodied intelligence, simulation-driven content creation, video understanding, and evaluation of physical competence in multimodal models. The paper releases a blog, code repository (github.com/mirros-lab/code-as-world), and project page (mirros-lab.github.io/code-as-world), and is released under CC BY-NC-SA 4.0.

Future Directions

• Closing the simulator gap. The stated limitation is that simulators cannot faithfully capture the full range of physical conditions; improving the fidelity of the executable world space and detecting when a hypothesis is only locally plausible remain open problems.
• Extending verification beyond current signals. The paper notes that candidate selection and refinement are driven by a verifier while results are measured by independent metrics, inviting stronger verification that better reflects mechanistic correctness rather than visual agreement.
• Scaling the reasoning variant. The 27B reasoning model reports the highest average on QuantiPhy among the proposed variants; the separate 27B protocol is deferred to Appendix B.5, leaving questions about how far the approach scales with model size and chain-of-thought supervision.
• Broadening downstream use. The paper lists physical data generation, quantitative supervision, verifiable reward construction, and broader physical reasoning as downstream uses; how far these extend beyond QuantiPhy-style metric questions is not reported.

Target Audience

Researchers and practitioners in computer vision, multimodal foundation models, and embodied AI who work on physical reasoning, world models, simulation-grounded learning, or video understanding. It will also interest engineers building data generation or training pipelines that need verifiable physical supervision, and readers interested in how agentic search loops can be used to invert observations into executable hypotheses. A working knowledge of VLMs, reinforcement learning from verifiable rewards, and simulation-based scene representation is needed to get full value from the technical sections.

Authors’ abstract

Physical understanding and reasoning depend on forming compact and generalizable representations of the world. While modern vision-language models can recognize and explain diverse physical events, they often lack explicit representations of the underlying mechanisms-such as object states, physical parameters, and governing dynamics-needed for reliably reasoning how the world evolves and responds to interventions. In this work, we introduce Code-as-World, a paradigm that represents physical worlds through executable world representations. By expressing physical composition, dynamic evolution, and visual appearance as executable code, Code-as-World provides a compact, quantitatively grounded, and controllable abstraction of the physical world. To construct such representations from multimodal observations, such as natural-language descriptions or real-world videos, we develop an agentic discovery loop inspired by abductive reasoning, where an agent proposes, executes, renders, verifies, and iteratively refines executable world hypotheses. As a concrete application, we use verified executable worlds to provide scalable physical supervision for training vision-language models on quantitative physical reasoning. Experiments show that Code-as-World-VL achieves state-of-the-art performance on QuantiPhy and surpasses leading proprietary models, highlighting the potential of executable world representations as a scalable foundation for physical intelligence.