WorldStream: Streaming Interactive 3D Worlds with Hierarchical Planning and Persistent State

AI summary

Overview

Research area: Generative world models and interactive 3D content creation, combining hierarchical planning with procedural/asset-driven geometry and a streaming runtime (computer vision and graphics systems).

Technical level: Advanced — the paper is written as a systems paper with formal notation for world state, planning objectives, and scheduling; the core ideas are accessible, but the evaluation and formalisms assume familiarity with world models and procedural generation.

Scope (one sentence): WorldStream is an open-ended 3D world model that generates playable terrain beyond a fixed map, grounds open-ended language in the running world, and restores visited regions together with their edits, all under bounded per-frame cost.

What This Paper Is About

Learned world models produce responsive frames from user actions, but their state lives in frames and latents, so a region dropped from memory can only be resynthesized and individual objects cannot be referenced later. Procedural and code-driven 3D systems keep every object inspectable and editable, but they build a fixed scene before play begins and do not decide what part of an unbounded world to prepare next. WorldStream's goal is to satisfy extent, grounded interaction, and reproducibility at the same time by separating a large decision model that plans semantics from deterministic local code that builds geometry, with both operating on one persistent world record.

Key Contributions

1. Dual-frontier streaming under bounded residency. A semantic frontier plans connected regions ahead of movement, while a narrower realization frontier compiles only the geometry needed soon; direction-aware admission bounds both stages without keeping the whole explored world in memory.
2. Geography-conditioned hierarchical planning with executable spatial contracts. Macro decisions organize multi-tile settlement, ecology, transport, and hydrological structure; Micro decisions specialize that contract into tile semantics under measured terrain and already-accepted neighbor commitments, and a joint resolution step converts typed probability distributions into feasible plans before any geometry is built.
3. Revisioned world state for durable interaction. A retained tile record holds plans, seeds, boundary commitments, stable identities, mutable actor state, and user edits beyond rendered geometry; it grounds current and future language actions, rejects stale asynchronous updates, and drives reconstruction on revisit.

Main Findings

• Hierarchy is necessary, and its two scales fail differently. On the same city-to-rural request, planning with local evidence only ("Micro only") stayed terrain-compatible but produced 0 roads, 0 buildings, 0 facilities — no settlement formed. Regional guidance without neighbor context ("No Micro-neighbor") kept centers and coarse roads but overbuilt, publishing 686 roads, 2,686 buildings, 16 facilities with a boundary jump of 0.361, versus 47 roads, 250 buildings, 13 facilities and a boundary jump of 0.089 for the full hierarchy. "Macro only" publishes no local content and is therefore omitted from the count table.
• Independent per-field selection invalidates most plans. On 100 retained probability distributions from the decision model, independent maximum-probability selection yielded valid plans only 37% of the time; per-constraint success was 55% (biome), 69% (terrain), 74% (Macro), 82% (suburban), 84% (landscape), and 88% (water). Joint resolution reached 100% on every constraint, gains of 2.70×, 1.82×, 1.45×, 1.35×, 1.22×, 1.19×, and 1.14×, without changing the model's probabilities.
• Directional prefetch, not semantic lookahead, prevents stalls. Over 100 route replays on a 4 km benchmark with a required 3×3 arrival neighborhood, planning ahead shortened recovery but left arrival coverage unchanged because construction still started too late. Directional prefetch brought almost the full neighborhood into the ready set before arrival and reduced total route waiting to 3.52 s; adaptive lookahead then reduced the remaining tail.
• Language grounding is strong, and errors concentrate on scope. Across 80 bilingual requests, the intended world operation or safe non-execution was recovered in 76 (95.0%). No-action requests scored 10/10 and unavailable targets or capabilities 10/10; actor/landmark was 19/20, clarification 19/20, and world editing 18/20, with residual errors mostly confusing a future-world instruction with a current edit.
• Consistency is enforced at the record level. Shared boundary records, a schedule-independent plan key, and revision stamps on background work make independently processed tiles agree regardless of arrival order or travel direction.
• Stated evaluation limits. Planning ablations report one complete run per variant and do not estimate model-sampling variance; prefetch replays measure backend tile readiness with browser installation excluded; the interaction benchmark measures interpretation against fixed world facts and does not isolate a complete edit–evict–revisit cycle.

Methodology in Plain English

The world is cut into fixed-size tiles, and each tile gets a persistent record: its geographic evidence (height, slope, water, surface), its accepted semantic plan, commitments it shares with neighbors along their common edge, explicit object state, user edits and future intent, a revision counter, and compatible generator and asset versions. The visible 3D geometry is treated as a disposable compilation of that record — eviction throws away geometry, and revisiting recompiles the same record to get the same place back.

Planning happens at two scales. A Macro planner looks at regional relief, drainage, shoreline, and the accepted land use and road ports of nearby tiles, and decides regional structure. A Micro planner then takes one tile, combines finer surface evidence with the covering Macro contract and its neighbors' accepted commitments, and emits an executable specification (semantic zones, boundary connections, facility and entity anchors, generator parameters). Crucially, the decision model outputs a preferred value plus a distribution over alternatives for each field; a bounded search then picks the joint assignment that satisfies terrain, water, biome, Macro-role, and adjacency constraints, so plans are known to be buildable before geometry exists.

Deterministic local code turns accepted plans into terrain, roads, buildings, vegetation, and entities, following spatial dependencies (terrain first, then circulation, then buildings, then vegetation). Vegetation is placed by combining an abundance parameter, a spatially coherent field, and local suitability into a proposal probability, then filtering by biome, slope, water masks, and spacing.

For streaming, the runtime classifies tiles as required, preferred, or speculative, predicts which tiles the player will enter and when, and admits work under a planning budget and a smaller construction budget with a shorter construction horizon. Language requests are interpreted against a compact view of the local world plus a list of available capabilities, then executed as a record update that is only committed if the tile's revision still matches — rejecting stale asynchronous work.

Why This Matters

Impact on research. The paper reframes interactive world modeling around what persists rather than what is rendered next, and shows a clean division of labor: a learned decision model handles semantics, deterministic code handles geometry, and a shared record makes both reproducible. Its ablation showing that structured model output alone leaves 63% of plans infeasible is a useful warning for any pipeline that treats typed generation as sufficient.

Real-world applications.

• Open-world games and simulations that must extend past authored maps while keeping world state stable across unload/reload.
• Embodied agents and training environments where terrain and stateful NPCs must persist and respond to natural-language direction.
• Interactive content creation tools where a user edits a region in language and expects the edit to survive after the region leaves memory.
• Large-scale procedural authoring of cities, coastlines, and ecologies where regional coherence across independently generated tiles matters.

Industry relevance. The design targets bounded per-frame and per-tile cost, worker-process compilation, and asset caching, which are the practical constraints for shipping a streaming world on a browser-based runtime (the evaluation uses a Three.js/WebGL runtime at 1920×1080 with three compile workers). The separation of planning from rendering also lets planning capacity and rendering capacity scale independently.

Future Directions

• Explicit scope representation. Most remaining interaction errors involved distinguishing a future-world instruction from a current edit, which the authors argue needs the same explicit representation as action and target.
• End-to-end eviction and revisit validation. The paper states its interaction benchmark does not isolate a complete edit–evict–revisit cycle, so measuring that loop directly is an open step.
• Sampling variance and cost reporting. Planning ablations use one run per variant and do not estimate model-sampling variance; broader repeated evaluation, plus per-frame cost measurements, would strengthen the claims.
• Adaptive lookahead and budget policy. Adaptive horizons tuned from recent dispatch-to-finish times reduced the residual tail; how far such adaptation can be pushed, and how it behaves under variable hardware, is left open.

Target Audience

Researchers and engineers working on generative world models, interactive 3D content creation, and procedural generation systems — particularly those building streaming runtimes where memory is bounded and world state must survive eviction. It is also relevant to game-technology and simulation teams evaluating whether to combine learned planners with deterministic geometry pipelines, and to readers interested in grounding language in persistent 3D state.

Authors’ abstract

We present WorldStream, an open-ended 3D world model with three simultaneous capabilities: it extends playable terrain beyond any pre-specified map, acts on open-ended language in the running world, and restores visited places with their edits. A hierarchical planner over a typed decision model emits structured decision state: Macro fixes settlement, ecology, transport, and hydrological structure across a group of tiles; Micro specializes that contract into tile semantics under measured terrain and accepted neighbor commitments. Deterministic local code compiles accepted plans into terrain, roads, settlements, vegetation, and entities. Streaming separates their reach: plans run ahead along the predicted corridor, compilation only where movement will soon need it, geometry behind the user released. One world record retains plans, seeds, object identities and mutable state, boundary commitments, and edits by attribute, revised as the user advances and edits; returning tiles recompile from it. The same record grounds interaction: the model acts on existing entities, including stateful NPCs that update as they act, edits the present region, or attaches intent to unplanned regions. Together they yield a world expanding under bounded budgets, connected across independently planned tiles, and interactive.