OctWorld: Long-Range World-Consistent Video Generation with Octree-Based 3D Mapping

AI summary

Overview

• Research area: Generative computer vision — specifically world-consistent, camera-controllable video generation and incremental 3D scene mapping.
• Technical level: Advanced. The paper assumes familiarity with diffusion models, autoregressive video generation, Truncated Signed Distance Function (TSDF) fusion, sparse octrees, and Marching Cubes.
• Scope: The paper introduces OctWorld, a video diffusion framework that pairs a chunk-wise autoregressive video generator with OctMap, a spatially adaptive octree-based 3D memory, to produce long-range, world-consistent video from a single image and a user-specified camera trajectory.

What This Paper Is About

Generating an explorable world from one image requires the model to keep the scene geometrically and visually consistent as the camera moves — and especially when it later returns to a region it already generated. Prior geometry-aware methods mostly stay within a single generation chunk and handle only short-term coherence; the few that add memory (point-cloud caches or stored RGB frames) lack principled fusion and drift over long trajectories. OctWorld's goal is to make long-range generation — long camera paths with wide viewing-angle coverage — spatially consistent by maintaining one persistent, globally fused 3D memory that is updated from every generated RGB-D chunk.

Key Contributions

1. The first framework enabling an extensible, globally fused 3D memory for long-range world-consistent video generation.
2. OctMap, a spatially adaptive octree-based TSDF memory that progressively fuses generated RGB frames and their associated depth observations into a global 3D representation, balancing geometric fidelity against memory efficiency in open-domain settings where scene scale is unknown and can evolve during generation.
3. Empirical results showing the model outperforms prior methods on existing benchmarks and on longer-range generation cases, plus a demonstration that OctMap outperforms both point-cloud caches and naive fixed-resolution TSDF volumes.
4. Two model variants: OctWorld-1.0 (built on FramePack, jointly generating RGB and depth) and OctWorld-1.5 (built on the Wan2.1 14B I2V backbone, with depth decoupled and supplied by an external streaming depth estimator).

Main Findings

• WorldScore benchmark: OctWorld attains a WorldScore of 50.31, ahead of Voyager (49.93), AETHER (46.59), Gen3C (45.48) and VMem (42.4). The paper reports that OctWorld achieves the highest scores in 3D Consistency and Photometric Consistency, ranking second in Camera Control and Style Consistency.
• Spatial memory on Re10K: On 80 of the longest Re10K test videos (approximately 120-frame sequences extended by an exact reversed trajectory), OctWorld records LPIPS 0.1633 and Chamfer Distance 0.0345, versus Gen3C (0.1933 / 0.0457) and VMem (0.4010 / 0.0620).
• User study: With 22 participants over 17 open outdoor scenes and 433-frame trajectories, 53.8% preferred OctWorld, 39.5% Gen3C, 0.9% VMem, and 5.8% were unsure.
• Ablation on world memory: With 36 inference frames per chunk, OctWorld scores LPIPS 0.1412 / CD 0.0252, beating fixed TSDF (0.1996 / 0.0345), point cloud (0.2749 / 0.0916 at f=16, and 0.2509 / 0.0556 for TSDF at f=16), and the no-OctMap variant (0.2128 / 0.0534).
• Noise augmentation trade-off: Noise level 0 gives the best spatial metrics (LPIPS 0.1412, CD 0.0252) but weaker 3D Consistency (67.18); level 3 (0.1633 / 0.0274) raises 3D Consistency to 71.67 and WorldScore Static to 50.31; level 5 degrades to 0.1806 / 0.0340 with 71.09. Level 3 is used in all experiments.
• Reprojection rendering quality (100 SpatialVID scenes): OctMap achieves PSNR 23.47, SSIM 0.74, LPIPS 0.27 with 7.26 GB memory and 17.94 s processing time. Point-cloud fusion scores 19.89 / 0.57 / 0.38. Fixed TSDF improves as voxels shrink (20.47 → 22.08 PSNR from voxel size 0.007 to 0.002) but memory grows to 9.66 GB, and voxel size 0.001 ran out of memory.
• Efficiency: One diffusion step takes 3.6 s on a single NVIDIA A100. OctMap memory ranges from 1 to 10 GB depending on scene scale, with RGB-D integration at approximately 4–6 FPS. Across 17 sequences of 505 frames each, OctMap averages 7.36 s integration and 4.60 s mesh extraction per 36-frame chunk, peaks at 22.93 GB CUDA memory, and ends with 34,659K allocated octree nodes.
• Model size comparison: OctWorld uses 14.50B parameters and runs at 0.136 FPS with 51 GB runtime memory, compared with Voyager (13.31B, 0.041 FPS, 48 GB), Gen3C (7.24B, 0.117 FPS, 41 GB), Aether (5.57B, 0.223 FPS, 34 GB) and Vmem (1.26B, 0.400 FPS, 19 GB).

Methodology in Plain English

The system generates video one chunk at a time. Given a starting image and a camera path, it estimates depth for that image, integrates it into a 3D memory, and then repeats a loop for each chunk of the trajectory: extract a mesh from the current memory, render that mesh into the target camera views (with visibility masks), feed those renderings plus a compressed pack of history frames plus explicit camera embeddings into the diffusion model, generate the RGB chunk, obtain its depth (either predicted jointly or from an external streaming depth estimator), and fuse the new RGB-D observation back into the memory.

The distinctive piece is OctMap. It is a TSDF volume — the standard volumetric fusion representation from SLAM — but instead of a fixed voxel grid it lives in a sparse octree that grows as evidence arrives. Scene scale in open-world generation is unknown and changing, so a fixed resolution either wastes memory or blurs detail. OctMap solves this by choosing a voxel size per pixel based on depth: far-away geometry gets coarse voxels, close-up geometry gets fine ones. Concretely, for each pixel the desired voxel size is z divided by the square root of the product of the focal lengths, and the octree depth is chosen as the finest level whose voxel size is no larger than that. Additional samples are taken along each camera ray within the truncation margin, the octree is traversed from the root to the required depth, and uninitialized nodes are allocated and split, with children inheriting parent attributes by spatial interpolation. TSDF values and colors are then fused with the standard weighted running average, using a truncation margin proportional to the local voxel size. Leaf-node cubes are meshed in parallel with Marching Cubes, and duplicated vertices at shared corners are merged.

The generator itself is built on a pretrained video diffusion backbone with long-horizon context modeling. History frames are packed into a fixed-length latent context using the backbone's context compression strategy. Depth, when predicted jointly, is normalized by the maximum depth of the history frames and represented as a grayscale image so it fits the 3-channel video encoder, with depth latents concatenated to RGB latents along channels. Cameras are normalized to the last history frame's coordinate frame and turned into Plücker ray embeddings processed by a lightweight convolutional adapter. During training, a randomly sampled noise level is applied to the memory and temporal context to close the train-inference gap that arises because, at inference, history consists of the model's own imperfect predictions.

Training data comes from RealEstate10K, DL3DV, and a static subset of SpatialVid, filtered with dataset annotations and a vision-language model to remove dynamic clips and clips with strong illumination changes or scene transitions, with depth, confidence and camera trajectories estimated by Pi3. Training uses FramePack F1 (14B) with the Hunyuan DiT backbone, bringing the total to 15B parameters, trained with AdamW at a learning rate of 2×10⁻⁵ at 36×480×832 resolution — first 80K steps at a reduced 16×480×832 temporal resolution, then 30K steps at full resolution — on 32 NVIDIA H100 GPUs with a total batch size of 64.

Why This Matters

The work reframes world memory in generative video as an incremental mapping problem rather than a retrieval problem, borrowing the fusion machinery of SLAM while relaxing the SLAM assumptions (known metric scale, bounded extent) that do not hold for open-domain generation. That bridge is the paper's main intellectual contribution, and it yields measurable gains over point-cloud caches and fixed-resolution TSDF on both generation quality and memory cost.

Real-world applications:

• Virtual production and film previsualization, where a director explores a set along a scripted camera path and revisits locations.
• Game and immersive-environment prototyping from a single reference image, enabling free-viewpoint exploration of scenes that do not exist yet.
• Virtual tours and real-estate visualization (the evaluation data itself comes from RealEstate10K), where long trajectories retracing rooms are the normal usage pattern.
• Simulation and data generation for embodied AI and robotics, where long-range geometric consistency is a prerequisite for usable training environments.

Industry relevance: the framework builds on existing open backbones (FramePack, Wan2.1 14B, Hunyuan DiT components), and the reported 3.6 s per diffusion step on an A100 with 1–10 GB of OctMap memory indicates the memory module is not the bottleneck. The authors note the backbone dominates inference time, so the approach stands to benefit directly from progress in real-time video generation.

Future Directions

• Dynamic content. Training data predominantly features static scenes, and the pipeline explicitly filters out dynamic clips; extending OctMap and the generator to moving objects is not addressed.
• Real-time exploration. The paper points to voxel pruning and efficient octree querying as a path to accelerating OctMap toward real-time interaction.
• Truly unbounded scale. OctMap adapts resolution dynamically, but how the canonical root volume, memory ceiling, and drift behave under arbitrarily large or endlessly expanding environments remains open.
• Robustness of the depth source. OctWorld-1.5 decouples depth into an external streaming estimator; how sensitive long-range consistency is to that estimator's errors, compared to the jointly trained OctWorld-1.0, is only partially explored.

Target Audience

Researchers and graduate students working on video diffusion models, world models, novel-view synthesis, and 3D scene representation, as well as graphics and SLAM practitioners interested in how hierarchical volumetric fusion can be adapted to generative settings. Practitioners building explorable or interactive environments from single images will also find the memory-efficiency and long-trajectory results directly relevant.

Authors’ abstract

We present OctWorld, a video diffusion framework with persistent 3D memory for generating explorable, world-consistent, and high-fidelity visual scenes. Given a single image, OctWorld performs stable autoregressive world generation along user-specified camera trajectories. We focus on long-range generation, characterized by extended camera paths and wide viewpoint coverage, where preserving spatial consistency is particularly challenging when previously generated regions are revisited. To address this problem, we introduce OctMap, an extensible and spatially adaptive 3D memory that progressively fuses generated visual observations and their corresponding depth maps into a global representation. OctMap employs TSDF fusion within a dynamic sparse octree whose spatial resolution adapts to image evidence. This design preserves geometric and appearance details across diverse scene scales while maintaining low memory overhead. Experiments demonstrate that OctWorld generates long-range, spatially consistent videos and outperforms prior methods on both existing benchmarks and challenging long-range generation settings. OctMap also provides clear advantages over point-based caches and fixed-resolution TSDF volumes. Project page: https://maxtirerror.github.io/octworldpage/