InfiniteDiffusion: Bridging Learned Fidelity and Procedural Utility for Open-World Terrain Generation

AI summary

Overview

Research area: Generative computer vision and computer graphics, specifically diffusion-model-based procedural terrain generation (SIGGRAPH Conference Papers '26, Los Angeles, July 19–23, 2026; DOI 10.1145/3799902.3811080; arXiv:2512.08309v4 [cs.CV] 03 May 2026).

Technical level: Advanced. The paper assumes familiarity with denoising diffusion, consistency models, latent diffusion, VAE-style autoencoders, FID, and fBm/Perlin noise.

Scope: One sentence: the paper introduces a training-free way to run diffusion sampling over an infinitely large domain while preserving seed-consistency and constant-time random access, and applies it to build a real-time learned procedural terrain generator.

What This Paper Is About

For decades, procedural worlds have been built with noise functions like Perlin noise, which are infinite, seed-consistent and fast, but look smooth and lack the hierarchical structure of real geography. Diffusion models produce far more realistic structure, but they are normally trapped inside a fixed, bounded canvas. This paper closes that gap: it reformulates diffusion sampling so that an unbounded image can be lazily queried region-by-region, then uses that reformulation to generate entire planets of realistic terrain at interactive rates.

Key Contributions

1. InfiniteDiffusion, a training-free generalization of MultiDiffusion (Bar-Tal et al., 2023) to infinite domains. It reformulates the sampling update so only the windows overlapping a queried region R are evaluated, defined via a function κ mapping a region to its overlapping window indices. The paper claims three retained properties of procedural noise: seamless infinite extensibility, seed-consistency, and constant-time random access, plus parallel window evaluation (formal proofs in Appendix B).

2. An open-source infinite-tensor framework, a Python library for constant-memory streaming and composable manipulation of unlimited tensors. It exposes implicitly infinite tensors with a PyTorch-like interface, keeps only a visible region and bounded transient cache in memory, and supports tiled disk storage.

3. Terrain Diffusion, described as the first practical learned procedural terrain generator, built from a global elevation dataset, a signed square-root elevation transform, a compact Laplacian encoding, and a hierarchy of diffusion/consistency models with an EDM2 backbone modified as in sCM.

4. Validation on both a general text-to-image model and terrain. InfiniteDiffusion is measured against bounded MultiDiffusion on Stable Diffusion across 8 prompts, and Terrain Diffusion is evaluated for FID, tiling behavior, and latency, including an integration that replaces Minecraft's native world generator.

Main Findings

• Truncating the step count is what makes infinity tractable. On a Stable Diffusion reference dataset across 8 prompts, InfiniteDiffusion with T=1 scores FID 7.20 and T=2 scores 5.83, against 5.75 for bounded MultiDiffusion (effectively T=50 in the comparison) and 1.94 for Stable Diffusion itself. T=1 retains overall structure but introduces artifacts in many cases; T=2 effectively saturates FID and removes most artifacts. Appendix D reports that T=5 eliminates practically all artifacts across every scenario tested, with diminishing overhead for larger regions.

• Perlin blending fails badly on terrain statistics. FID-50k results: Perlin Blending 186.70, Naive Tiling 74.44, Naive InfiniteDiffusion (T=0) 27.61, InfiniteDiffusion (T=1) 19.78, InfiniteDiffusion (T=2) 14.78, while non-tiled distilled output is 12.72 (the stated theoretical lower bound) and non-tiled non-distilled output is 8.11.

• Two latent blending steps nearly recover untitled fidelity. InfiniteDiffusion with T=2 preserves base consistency-model fidelity at 14.78 versus 12.72, scaling from finite training crops to infinite worlds with little quality degradation.

• Laplacian encoding is a large stabilizer. Because normalization makes small σ=0.01 errors correspond to ±25 m after denormalization, the paper predicts a low-frequency plus residual representation; residual errors are over 30× smaller in magnitude. The Laplacian denoising step reduces FID of the untiled core diffusion model from 21.51 to 8.11, and of the consistency model powering the final pipeline from 75.15 to 12.72. The re-extracted low-frequency channel satisfies L ≈ L̂ even under strong synthetic noise.

• Signed square-root transform equalizes variance across tiles. It decreases the correlation between mean and log standard deviation of tiles from 0.66 to 0.31, allowing training with a more focused noise distribution and improving visibility of small features such as coastlines.

• The window weight shape matters. A separable linear weight window decaying from 1 at the center to ε at the boundary improves FID from 19.32 to 14.78 compared to a constant map.

• Latency supports interactive use on a single consumer GPU. On one NVIDIA RTX 3090 Ti: InfiniteDiffusion (T=2) has time to first tile (TTFT) 1.72 ± 0.19 s and time to second tile (TTST) 0.66 ± 0.18 s; T=1 gives 1.39 ± 0.20 s and 0.63 ± 0.16 s; Naive InfiniteDiffusion gives 2.05 ± 0.22 s and 1.60 ± 0.17 s; Independent Tiles gives 0.51 ± 0.05 s and 0.22 ± 0.02 s. Averages and standard deviations come from 1000 runs at random locations.

• Speed relative to traversal. An F-35 at roughly 550 m/s would cross a 512×512 tile at 90 m resolution in about 84 seconds, during which Terrain Diffusion can produce 130 additional tiles. At a 1:15 miniature world scale, a 60 mph vehicle effectively encounters terrain at 405 m/s, and the system holds a 170× performance buffer. At orbital velocity (≈7,700 m/s) generation remains 9× faster than traversal.

• Tiling steps drive the cost. TTFT benefits most from larger T; TTST is similar between T=1 and T=2. Independent tiling is about 3x faster than InfiniteDiffusion, roughly matching 50% stride in the latent model (4x overhead) and 75% stride in the decoder (1.78x overhead). A minimal cache of approximately 10MB achieves optimal TTFT and TTST by memoizing only windows needed for the previous query.

• Outputs are visually coherent. Figure 4 shows 20 1024×1024 tiles from the same seed used in Figure 1, with sharp ridges, coherent river basins, smooth transitions and no visible tiling artifacts. Figure 1's leftmost panel spans roughly five million square kilometers with about 2.2 million square kilometers of land area, comparable to the Congo, and the panel sequence spans four orders of magnitude in scale.

• In-engine streaming works. Terrain Diffusion replaces Minecraft's native world generator, routing elevation and biome queries through the model and mapping climatic outputs to biomes with a lightweight rule set; runtime is dominated by Minecraft's own generation logic rather than the model. Heightmaps are bilinearly upsampled 4× for these interactive visualizations.

• Scaling with cascade depth. Panning by S pixels at the finest layer moves only S/a pixels at the next coarser layer, so the amortized cost per S×S region is O(S²)(1 + 1/a + 1/a² + ⋯) = O(S²)·a/(a−1), independent of cascade depth, and just 4/3 ≈ 1.33× the cost of a single layer for 4× super-resolution at infinite depth.

Methodology in Plain English

MultiDiffusion generates images larger than a model's training resolution by running the model on many overlapping windows and averaging the predictions with per-window weight maps. The catch is that every window must sit inside a fixed, finite canvas, and every window must be evaluated to complete one step.

The paper removes that canvas. Regions are redefined as any rectangular subset of the integer grid, the window index set becomes countably infinite (S = Z² in the experiments), and each window is a sliding square of side H = W with stride s, so window (i,j) covers [is, is+H) × [js, js+W). Because the MultiDiffusion update would now be an infinite sum, the authors instead evaluate it lazily: they define κ(R) as the finite set of windows overlapping a queried region R and only evaluate those windows. Each timestep keeps two sparse infinite tensors — a numerator Aₜ accumulating Wᵢ ⊗ Φ(Jₜ₊₁[Rᵢ] | yᵢ) and a denominator Bₜ accumulating Wᵢ — so that Jₜ[R] = Aₜ[R]/Bₜ[R]. Tensors are stored as per-window contribution pairs, with a lightweight in-memory LRU cache, where evicting a pair is safe as long as its index is also removed from the processed-window set Pₜ (it is then simply recomputed on the next query).

The remaining obstacle is that a query at step t usually needs a larger region at step t+1, so large T becomes prohibitively expensive, especially for the first query. The fix is to redefine Φ as an arbitrary denoising function — a few-step consistency model, or a short sequence of standard diffusion steps — which decouples T from the internal diffusion schedule. This is why T=1 or T=2 suffices.

For terrain, the authors first build a global dataset by merging 90m MERIT DEM land topography with ETOPO1 ocean bathymetry and WorldClim climatic data, processing it into equal-area tiles by stretching longitude with latitude, splitting into 2048×2048 tiles, with 80% randomly assigned to training and the remaining 20% held for validation. Elevations are signed-square-rooted to reduce variance spread. Generation runs through a hierarchy: a coarse planetary model produces basic structure at 23km/pixel from a rough procedural or user-provided layout; a core latent diffusion model turns that into realistic 46km tiles, predicting a 64×64 low-frequency elevation channel plus a latent map for a 512×512 patch at 90m resolution (latents come from a separately trained VAE-style autoencoder whose decoder is discarded); and a diffusion decoder expands the latents into high-resolution residuals, with nearest-neighbor interpolated latents concatenated to the noisy input at each step. The core model is conditioned on 4×4 patches of elevation (each patch pixel about 23km, with prediction covering the 2×2 interior), using three channels — mean elevation, 5th percentile elevation, and a binary data-availability mask — plus tile mean temperature, temperature variation, annual precipitation and precipitation seasonality, with missing ocean climate values replaced by a standard gaussian. Laplacian denoising cleans the low-frequency channel by decoding, downsampling (8×8) and blurring (σ=5) to re-extract L̂, with high-frequency noise redirected to the residual, so final synthesis uses L̂ + H. Except for the coarse model, all diffusion models are distilled into continuous-time consistency models, and AutoGuidance is applied for extra fidelity.

Why This Matters

Impact on research. InfiniteDiffusion is training-free, so it can be layered onto existing pixel- or voxel-based diffusion models rather than requiring a new generative model. It also reframes a trade-off the paper says existing approaches face: MultiDiffusion scales exponentially in compute for ultra-high resolution, while cropping-based methods scale linearly but discard context at every step. Lazy evaluation over infinite tensors offers a third option — compute only the queried region while keeping global coherence — and the constant-time random access plus seed-consistency properties (order-invariant outputs, unlike autoregressive tiling methods such as BlockFusion and WorldGrow) make it a different foundation for interactive worlds than sequential tile generation.

Real-world applications

• Open-world game world generation, demonstrated concretely by swapping out Minecraft's native generator and streaming terrain during arbitrary traversal.
• Planetary-scale simulation and geospatial visualization, where a single seed can produce a seamless global elevation field across four orders of magnitude of scale.
• Synthetic environments for simulation and training, where terrain is generated on demand at interactive rates instead of being authored or stored.
• Film, VFX and virtual production, where a director could zoom into arbitrary detail of an implicitly ultra-high-resolution environment while only computing the queried region.

Industry relevance. The system targets consumer hardware — a single NVIDIA RTX 3090 Ti — with warmup of 1.72 seconds and steady-state generation of 0.66 seconds for the highest quality configuration (T=2), and it exposes a procedural-noise-like interface rather than a fixed baked asset. The open-source infinite-tensor framework is offered as a reusable runtime layer for streaming and composable manipulation of unbounded tensors, which is directly relevant to engine and tooling developers. The paper frames procedural noise as the incumbent technology, so the relevant industry contrast is against noise functions whose latency is in microseconds but whose fidelity is limited.

Future Directions

• Extending the terrain hierarchy with additional variables as outputs, conditioning, or both — soil properties, other climatic variables, or satellite imagery — for greater control and new downstream applications.
• Applying the InfiniteDiffusion algorithm to additional dimensions, moving into voxel-based 3D synthesis rather than 2D heightmaps.
• Pushing ultra-high-resolution generation further using lazy evaluation and deep cascades, on the argument that amortized cost stays independent of cascade depth (1.33× a single layer for 4× super-resolution at infinite depth).
• Addressing the stated limitations: artifacts from T=2 on more ambiguous prompts (usually resolved at T=5, at some TTFT cost), inefficiency for queries of much smaller scattered regions when uncached — which leaves Minecraft features relying on long-range biome searches (specifically /locate biome and explorer maps) unsupported — and the difficulty of acquiring training data at the coarsest layer, where synthetic data is suggested as an option for distilling other models or simulations into approximate procedural counterparts.

Target Audience

Graphics and computer-vision researchers working on diffusion models, infinite or ultra-high-resolution generation, and procedural content; game engine and world-generation engineers who need terrain that streams rather than loads; technical artists and simulation engineers building large virtual environments; and geospatial or Earth-science practitioners interested in learned planetary-scale elevation models. Readers without a background in diffusion sampling and consistency distillation will find the algorithmic sections difficult, though the terrain pipeline and results are described in mostly architectural terms.

Authors’ abstract

For decades, procedural worlds have been built on procedural noise functions such as Perlin noise, which are fast and infinite, yet fundamentally limited in realism and large-scale coherence. Conversely, diffusion models offer unprecedented fidelity but remain generally confined to bounded canvases. We introduce InfiniteDiffusion, a training-free algorithm that reformulates diffusion sampling for lazy and unbounded generation, bridging the fidelity of diffusion models with the properties that made procedural noise indispensable: seamless infinite extent, seed-consistency, and constant-time random access. To demonstrate the utility of this approach, we present Terrain Diffusion, a framework for learned procedural terrain generation with a procedural noise-like interface. Our framework outpaces orbital velocity by 9 times on a consumer GPU, enabling realistic terrain generation at interactive rates. We integrate a hierarchical stack of diffusion models to couple planetary context with local detail, a compact Laplacian encoding to stabilize outputs across Earth-scale dynamic ranges, and an open-source infinite-tensor framework for constant-memory manipulation of unbounded tensors. Together, these components position diffusion models as a practical foundation for the next generation of infinite virtual worlds.