InfiniDream: Open-Ended Photorealistic 3D Worlds Through Procedural Generation

AI summary

Overview

Research area: Computer vision and computer graphics, specifically text-driven 3D asset and 3D world generation built on procedural (code-based) generators.

Technical level: Advanced. The paper assumes familiarity with vision-language models (VLMs), procedural modeling systems such as Infinigen, and standard 3D-generation evaluation metrics (CLIP, SigLIP2, DINOv3, ImageReward).

Scope: InfiniDream is a VLM-driven framework that extends an existing procedural 3D codebase beyond its predefined asset factories and composes the results into large-scale, editable, photorealistic 3D worlds from text prompts.

What This Paper Is About

Procedural generation (building 3D content from executable programs) gives editable, diverse, seed-controllable assets, but existing photorealistic procedural systems are limited to hand-authored asset factories — Infinigen, for example, is built on over 40,000 lines of code with predefined categories. Meanwhile, VLM-based code generation can produce novel content but yields relatively simple geometry, and VLM-guided procedural methods are confined to calling predefined functions and adjusting parameters. The paper asks whether a VLM can instead extend a sophisticated procedural codebase to create genuinely new, photorealistic assets, and then arrange those assets into large, semantically correct 3D scenes.

Key Contributions

1. Symbolically Constrained Editing (SCE) — a method that turns a large procedural codebase into a manageable editing space by extracting key code structures and explicit editing constraints, so a VLM can create novel photorealistic assets beyond predefined factories while preserving editability and seed-based diversity.
2. An end-to-end text-to-world pipeline — combining semantic scene planning by a "Director" with Deterministic Frustum-Guided Grid Placement, which converts semantic placement constraints into precise 3D coordinates without additional VLM calls or render-based verification.
3. Scalable population and animation — semantically budgeted asset pooling (generating K prototypes and cloning them into N instances), integration with Infinigen's fine terrain pipeline, and procedural animation combining World Motion and Local Animation.
4. Comprehensive validation — object-level (100 tasks, six categories) and scene-level (20 tasks) benchmarks across four perceptual metrics, a diversity analysis against TRELLIS.2, object- and scene-level ablations, and a 30-participant human preference study.

Main Findings

• Best object-level scores across all four metrics. On 100 tasks across six categories, InfiniDream reaches CLIP 30.07, SigLIP2 0.148, DINOv3 0.554, and ImageReward -0.62, versus the strongest baseline VLM (Harness) at 29.20 / 0.139 / 0.498 / -0.82, and Infinigen at 22.84 / 0.079 / 0.253 / -1.90. Code executability is 98%, second only to VLM (Harness) at 99%.
• Best scene-level scores across all four metrics. On 20 scene tasks, InfiniDream scores CLIP 29.64, SigLIP2 0.143, DINOv3 0.361, and ImageReward -0.877, compared with VLM (Harness) 28.85 / 0.126 / 0.321 / -0.894, Infinigen 25.21 / 0.079 / 0.182 / -1.710, and Code2Worlds 22.76 / 0.072 / 0.114 / -2.014.
• Strong human preference. Across 30 participants on 10 object and 10 scene tasks, InfiniDream's preference rates are 81.3% (alignment) / 73.7% (quality) / 77.5% (overall) at the object level and 91.0% / 75.3% / 83.2% at the scene level.
• More visual diversity than TRELLIS.2 at comparable semantic alignment. Over 15 prompts with five variations each, InfiniDream achieves a higher average pairwise DINOv2 feature distance (0.6704 vs 0.6084) with a slightly lower SigLIP2 score (0.1225 vs 0.1317).
• Every component of SCE matters. On 25 animal and plant tasks, the full pipeline scores CLIP 30.37 / SigLIP2 0.1417 / DINOv3 0.494 / ImageReward -0.580, ahead of removing the reference image (28.68 / 0.1338 / 0.363 / -0.978), random factory selection (27.39 / 0.1157 / 0.296 / -1.520), removing the Planner (28.72 / 0.1274 / 0.368 / -1.023), removing the Refiner (28.80 / 0.1192 / 0.334 / -1.420), and limiting refinement to three rounds (29.33 / 0.1354 / 0.438 / -0.824). Notably, an unconstrained agent given the full Infinigen codebase and Bash access (Agent (Infinigen)) reaches only 29.01 / 0.1343 / 0.387 / -0.968, below the full constrained pipeline.
• Deterministic grid placement sharply reduces unreadable objects. Over 20 scenes, the full method yields a 7.8% Unreadable Rate and Contact Spread P95 of 0.24 m, versus 27.4% / 0.25 m without grid search (direct 3D coordinate prediction), 40.3% / 0.74 m without anchoring, 14.0% / 0.38 m without semantic targets, and 50.9% / 0.75 m for a VLM placement agent. Ground-contact error P95 is 0.15 m for the full method, matching the w/o Grid Search and w/o Semantic Target variants.
• Clear margin on the subset Code2Worlds supports. On 29 of the 100 object tasks that Code2Worlds can handle, InfiniDream scores CLIP 30.24, SigLIP2 0.153, ImageReward -0.35, and DINOv3 0.583 versus Code2Worlds at 22.56, 0.080, -1.94, and 0.281.
• Capability coverage. In the paper's capability table, InfiniDream is the only listed method marked as supporting all eight criteria (novel procedural assets, seed diversity, procedural editing, text-to-scene, placement reasoning, semantic anchoring, scalable population, animation).

Methodology in Plain English

The system starts from a text prompt (and optionally a reference image) and works at two levels.

Asset level — Symbolically Constrained Editing. Instead of handing a VLM the whole Infinigen codebase (which the authors show leads to imprecise edits, and which an unconstrained coding agent handles worse than their constrained pipeline), they build a "Factory Brief" for the retrieved factory. It contains a Code Structure Map of key functions and dependencies, an "Editable Surface" of asset-specific modules found through dependency analysis limited to depth two, and read-only "Linked Infrastructure" for shared low-level algorithms. A Planner then writes a structured Editing Plan naming the strategy — Tweak (adjust parameters and high-level logic), Create (add new procedural functions), or Promote (selectively open protected modules) — plus the target locations, edits, dependencies, and risks. A Coarse Constructor executes the plan while keeping the original entry points and function signatures intact, so the population pipeline and random seeds still work; randomness is preserved by widening sampling ranges rather than hard-fixing values to match a reference image, and a vision-based Reviser iteratively compares renders to the target for up to six rounds, with a Selector picking the best candidate.

Scene level — semantic planning plus deterministic placement. A Scene Picker retrieves a suitable initial coarse scene from a library of 70 pre-generated environments (14 biomes with five seeds each). A Director produces a natural-language plan converted into a structured Scene Plan with add/replace/delete operations and semantic constraints, choosing among three placement modes (Hero for significant objects, Scattering for large-area distributions, Swarm for collective entities) and seven anchoring types (rooted, upright, tilted, conform, afloat, airborne, submerged), plus density, coverage, group size, instance count N, prototype count K, world motion, and local animation. Crucially, the VLM only makes discrete semantic decisions; all continuous geometry is computed deterministically. The camera's image plane is discretized into a 9×5 grid (17×9 when there are six or more instances), and candidate 3D points are found along camera rays — via ray-scene intersection for surface-supported objects, or at depth intervals (scaled ×1.3, up to 80 m) for airborne and submerged ones. Each candidate must pass five deterministic checks: visibility, collision against motion swept envelopes, inter-instance spacing, surface policy (type, slope limits, full support), and a projected-area band (preferred minimum 1.2%, relaxed minimum 0.2%, absolute maximum 25%). The nearest valid candidate wins, giving implicit size-aware placement; constraints relax progressively if nothing passes. Finally, asset pooling generates K seed-diverse prototypes and clones them into N instances, terrain refinement is applied, and animation is added before rendering.

Why This Matters

Impact on research. The paper reframes VLM-guided procedural generation from orchestrating existing factories to extending them under explicit symbolic constraints. The ablation result that a constrained pipeline beats an unconstrained coding agent with full codebase and shell access is a concrete data point for how to structure tool use over large codebases.

Real-world applications

• Synthetic training data: photorealistic, seed-diverse, automatically annotated 3D scenes for perception and embodied-AI training.
• Game and virtual-world development: editable procedural assets and large worlds generated from text, with parameter-level control rather than baked meshes.
• Simulation and digital twins: terrain-aware object placement with validated ground contact and collision envelopes.
• Pre-visualization and content pipelines: rapid text-to-scene prototyping that still exports editable programs.

Industry relevance. The work targets the practical gap between generative 3D tools that output uneditable meshes and procedural systems that require specialist programmers — a tradeoff that directly affects production pipelines in games, film, and robotics simulation.

Future Directions

• Animation generality. The authors state that the current animation pipeline relies on a predefined set of motion types and propose VLM-generated animation code for more complex, customized motions.
• Indoor environments. Extending InfiniDream to interiors with more complex spatial relationships and object interactions is named as future work.
• Broader factory coverage. The separate comparison on only 29 of 100 tasks (those Code2Worlds supports) highlights how factory coverage bounds comparisons; scaling SCE to more asset categories remains open.
• Reducing reliance on strong proprietary models. The pipeline uses Claude Opus 4.8 throughout and Gemini 3 Pro Image for reference generation, and the scene-level DINOv3 metric depends on a Gemini-generated reference image; how much of the result survives with weaker or open models is not reported.

Target Audience

Researchers and practitioners in 3D generative modeling, computer graphics, and procedural content generation; engineers building text-to-3D or text-to-world pipelines who need editable, seed-controllable output rather than meshes; and teams working on synthetic data generation or embodied-AI simulation environments. Readers should be comfortable with vision-language model prompting, Blender-based codebases, and standard image-text similarity metrics.

Authors’ abstract

Procedural generation plays a key role in synthetic data creation and game development, offering greater controllability, editability, and diversity than static mesh representations. However, VLM-based code generation often produces relatively simple 3D assets, while sophisticated photorealistic procedural systems require extensive manual development. Existing VLM-guided procedural methods remain largely constrained by predefined factories and APIs, limiting their ability to create novel asset types. We introduce InfiniDream, a VLM-driven framework for open-ended procedural 3D world generation. First, we propose Symbolically Constrained Asset Generation, which leverages existing procedural algorithms to create novel, photorealistic assets while preserving editability and seed-based diversity. Second, we develop an end-to-end text-to-world pipeline that combines semantic scene planning with Deterministic Frustum-Guided Grid Placement to construct large-scale 3D worlds with precise object placement, procedural population, and animation. Extensive evaluations on 100 object and 20 scene generation tasks demonstrate improvements across four perceptual metrics. Diversity analysis, ablation studies, and human preference evaluations further validate the effectiveness of our framework. Project page: https://yichuanh.github.io/infinidream-page/