Recursive Code World Models: Building Complex Worlds through Recursive Scene Programs

AI summary

Overview

Research area: Computer vision and 3D scene reconstruction, specifically code-based world models driven by vision-language coding agents.

Technical level: Advanced. The paper assumes familiarity with program synthesis, camera projection math, iterative render-and-compare loops, and standard image reconstruction metrics. The central idea, however, is intuitive enough to follow without the formalism.

Scope: The paper proposes Recursive Code World Models (RCWM), a framework that reconstructs a complex, executable 3D world as a nested scene program from a single reference image by applying the same construction solver recursively to subworlds.

What This Paper Is About

Existing code world models can represent a world as an executable program, but they do not specify how to build a complex one. A single reconstruction trajectory tends to either lose small details (when editing the whole scene) or break cross-part consistency (when refining parts in isolation). The authors' goal is a construction procedure that resolves fine-scale structure and preserves whole-scene geometry and spatial relationships at the same time.

Key Contributions

1. Recursive Scene Programs (RSP). An executable scene representation that organizes a complex world as compositional subworld programs with explicit child references, editable parameters, placement rules, and shared dependencies. Parents hold references to children, and a Compose operation swaps child programs in and out while preserving the surrounding code.

2. Recursive Code World Construction. A construction process in which the same complete solver is invoked at every level: establish the whole, recursively reconstruct unresolved parts, then revisit and refine their composition. This global–local–global cycle combines inherited observation geometry, visually guided recursive descent, and active parent-level refinement after children return.

3. Reference-aligned observation machinery. A paired cropping scheme (Crop on the reference image and CropCamera on the camera) that gives each child its own magnified reference view and matching camera projection, inheriting the parent's pose while preserving coordinates and magnification. Overlapping windows let the agent inspect object details, spatial relations, and contact boundaries.

4. Benchmark and ablation study. A parameterized Three.js compiler and evaluation against three image-to-scene-program baselines under the same frozen base model, plus ablations isolating construction order, recursion depth, and parent revisitation.

Main Findings

• Consistent gains over code-based baselines. RCWM achieves the highest PSNR and lowest LPIPS on all ten references, and the highest SSIM on nine of ten. On city-full, PSNR is 18.2 versus 15.5 (img2threejs), 14.5 (SEIG), and 9.5 (VIGA). On medieval-village, it reaches 18.7 PSNR versus 15.4, 10.9, and 10.5.

• Strong local-detail reconstruction. On park-lake, RCWM scores 23.3 PSNR, 0.99 edge F1, and 0.075 LPIPS — a large margin over the next best (18.4 PSNR, 0.92 edge F1, 0.169 LPIPS). On school-block and police-corner, edge F1 reaches 0.97 while baselines sit between 0.75 and 0.88.

• One exception on SSIM. On valley-village, VIGA obtains 0.41 SSIM versus RCWM's 0.38, although RCWM still leads on PSNR (14.1 vs 10.2), LPIPS (0.583 vs 0.743), and CLIP similarity (0.87 vs 0.68).

• Recursion depth matters, but not uniformly. On medieval-village, allowing free-depth recursion expands construction to three levels with 33 nodes and raises whole-frame PSNR from 16.8 to 19.0 and local SSIM from 0.52 to 0.60 compared with a fixed two-level recursion. On school-block, the free-depth run used only two levels and scored similarly to the fixed two-level variant.

• Baseline construction orders underperform. Flat + zoom, Local → global, and Global → local all fall short of the recursive variants. On medieval-village, Local → global degrades to 14.3 PSNR and 0.402 LPIPS, worse than plain flat editing at 16.4 PSNR.

• Programs generalize beyond the reference view. Rendering the delivered programs from azimuth rotations, raised orbits, and close-ups — without re-optimization — reveals coherent 3D geometry, with the caveat that crop-based scenes only contain content visible in their reference.

• Call trees confirm the recursion in practice. Logged call and return events show internal nodes following the complete whole–parts–whole cycle across multiple levels, with trees released for all ten scenes.

Methodology in Plain English

The system takes one image and produces a runnable 3D scene program (written for Three.js). A frozen vision-language coding agent does all the actual authoring; the paper's contribution is how that agent's work is organized.

The process is a single function F that calls itself. Each call does three things in order. First, it establishes the whole: the agent renders the current subworld inside its surrounding scene context, compares the render to the reference image, and edits the layout so that the overall arrangement and the interfaces between parts are right. Second, it prepares and recursively constructs the parts: it picks the children that still have unresolved detail, gives each one a cropped reference image and a matching cropped camera view at higher magnification, allocates each a slice of the remaining inference budget (holding some back for the parent's later review), and calls F on each of them. Third, it composes, refines, and returns: the returned child programs are spliced back into the parent, the assembled whole is re-rendered and compared to the reference, and cross-part problems — a sign covering the wrong window, a bridge that no longer meets its riverbanks — are corrected. The call returns only when a full review turns up no new actionable discrepancy, or when its budget runs out.

The key mechanics are inheritance and revisitation. Every child inherits its parent's camera pose and coordinate conventions through the crop operations, so local edits stay registered to the global frame. The parent's second look exists precisely because local refinements can introduce conflicts that no isolated child review would catch. The same cycle then repeats inside each child if it has unresolved structure of its own, producing a recursive call tree rather than a fixed scene-then-object pipeline.

Evaluation uses the same frozen base model (gpt-6-astra through Codex at high reasoning effort) for every method, with SEIG, VIGA, and img2threejs re-run under identical conditions. Metrics are PSNR, SSIM, Canny-edge F1 with three-pixel tolerance, AlexNet LPIPS, and CLIP ViT-B/32 similarity, computed against unmodified references at native resolution.

Why This Matters

Research impact. The paper reframes scene reconstruction as a control problem over computation rather than a single-pass generation problem. It argues that when a world contains more structure than one trajectory can resolve, the right move is to apply the entire solver — not just a decomposition step — to subworlds, and to treat the return to the parent as a first-class refinement stage. This connects image-to-code reconstruction to recursive inference work like Recursive Language Models, and distinguishes it from room-specific concurrent work such as FuncRoom-Agent by keeping recursion domain-general.

Real-world applications:

• Game and simulation asset pipelines that need editable, parameterized 3D scenes rather than baked meshes, starting from a single concept image.
• Digital twin and urban-planning tools where a rendered reference must be converted into a runnable, inspectable program with explicit spatial relationships.
• Robotics and embodied AI simulation, where scene programs can be handed directly to a physics engine for training or planning.
• Interactive content creation, where authors sketch or paint a view and receive an executable world they can edit by changing parameters or child components.

Industry relevance. The output is source code, not pixels or point clouds, which means it supports targeted editing, component reuse, version control, and integration with simulation — properties that matter for production pipelines. The finding that persistent, structured recursion beats single-trajectory editing also has direct implications for how agentic coding systems should allocate inference budget on long-horizon spatial tasks.

Future Directions

• Resolving single-view ambiguity. Hidden geometry, occlusions, and absolute scale remain undefined from one image. Multi-view input, depth cues, or explicit uncertainty modeling could let the recursion reason about what it cannot see.

• Controlling and predicting recursion depth. Deeper calls helped substantially on medieval-village but not on school-block. A principled criterion for when to descend — based on measured unresolved detail — would replace the current budget-driven behavior.

• Cost and convergence guarantees. Repeated refinement and cross-component interaction can consume substantial computation, and convergence is not guaranteed. Pruning, caching, or parallel scheduling of compatible branches are natural remedies; the paper notes that parallelism affects only scheduling, not which subproblems get solved.

• Scale and rigor of evaluation. The current study covers ten references, single runs per configuration, one base model, and ablations on two scenes, with no human evaluation of spatial relationships or editability. Larger benchmarks, repeated runs, and perceptual studies would strengthen the claims.

Target Audience

Researchers and advanced practitioners in 3D scene reconstruction, program synthesis, and agentic vision-language systems. It is most valuable to readers interested in code-as-representation approaches, recursive or hierarchical inference strategies, and render-and-compare feedback loops. Graphics engineers building editable asset pipelines and embodied-AI researchers needing simulation-ready scene code will also find the framework directly applicable, though they should expect to engage with the camera-cropping formalism and the budget-based solver design.

Authors’ abstract

Code world models represent worlds as executable programs, but this representation alone does not determine how to construct a complex world. We introduce Recursive Code World Models (RCWM), a framework for reconstructing complex 3D worlds in code from a single reference image. RCWM couples a Recursive Scene Program (RSP) representation with a construction solver that recursively calls itself. An RSP represents the executable world as compositional scene code, while each solver call follows the same complete process: establish the whole, recursively reconstruct unresolved parts, and revisit the whole to refine their composition. This global-local-global recursion gives fine-scale structures their own perception-and-editing loops while preserving scene-wide geometry and relationships. Reference-aligned views propagate a shared camera projection across levels, while parent revisitation addresses boundaries, spatial relations, and shared errors that emerge after local refinement. A vision-language coding agent directly compares reference images with scene renders to guide refinement, recursive descent, and return. Across complex scenes, RCWM outperforms prior code-based image-to-scene reconstruction methods. Ablation studies further support the benefits of recursive construction and suggest that deeper calls can improve finer-scale reconstruction. RCWM provides a recursive construction principle for building complex executable worlds from visual evidence.