Research
Persistent Planar Memory for Video World Models
Video world models that generate long camera trajectories need some memory of the 3D scene they have already produced, or they drift and forget structure once the viewpoint moves far from the first…

- arXiv
- 2610.23795
- Published
- 2026-10-16
- Authors
- Yuze He, Bin Tan, Zelin Gao, Yujun Shen, Yong-Jin Liu, Nan Xue, Yinghao Xu
AI summary
Overview
Research area: Computer Vision — video world models, autoregressive video diffusion, and 3D scene representation.
Technical level: Advanced. The paper assumes familiarity with diffusion transformers, video latent VAEs, camera-controlled generation, and multi-view geometry (depth, normals, plane primitives).
Scope: PPMem is a video world model that stores its 3D memory as a growing pool of oriented plane primitives, jointly predicted with each generated video chunk and re-rendered as depth conditioning for the next chunk.
What This Paper Is About
Video world models that generate long camera trajectories need some memory of the 3D scene they have already produced, or they drift and forget structure once the viewpoint moves far from the first frame. Existing 3D-grounded systems keep that memory as point clouds, surfels, TSDF volumes, or retrieved raw frames — representations that are either dense, externally estimated by a separate depth/reconstruction model, or not cross-view consistent. The paper asks whether a small set of oriented planes (walls, floors, facades) can serve as a compact, explicit, and self-predicted persistent memory for autoregressive video generation.
Key Contributions
- Plane primitives as world-model memory. The authors propose accumulating oriented plane primitives into a persistent plane memory via cross-chunk geometric merging, and argue this is the first such memory that is simultaneously compact, geometrically explicit, cross-view consistent, and adaptive.
- PPMem architecture. They instantiate the idea on a 5B Wan-2.2 diffusion transformer by adding a planar prediction head (adapted from Plana3R) and a Depth Token Fusion (DTF) module that re-injects plane-pool-rendered depth at token level, so a single framework outputs RGB video, metric depth, surface normals, and exportable planar geometry.
- Benchmark against 3D-grounded world models. They compare against Gen3C, Voyager, VMem, and SPMem on DL3DV and RealEstate10K in both short-context and long-context (300–400 frame) settings, plus ablations isolating the representation, the depth-conditioning mechanism, the decoder, and the loss terms.
Main Findings
- Short-range video quality (DL3DV, N=49): PPMem reaches PSNR 16.99, LPIPS 0.335, CLIP 0.909, DINOv2 0.861, versus Gen3C 15.74 / 0.474 / 0.847 / 0.710, SPMem 14.70 / 0.492 / 0.853 / 0.703, Voyager 12.84 / 0.568 / 0.796 / 0.615, and VMem 11.88 / 0.636 / 0.781 / 0.477.
- Short-range video quality (DL3DV, N=121): PPMem reports 13.98 / 0.464 / 0.863 / 0.769, against Gen3C 13.59 / 0.565 / 0.798 / 0.585 and SPMem 13.03 / 0.607 / 0.788 / 0.536.
- Short-range video quality (RealEstate10K): PPMem reports 22.46 / 0.168 / 0.965 / 0.949 at N=49 and 19.34 / 0.254 / 0.952 / 0.910 at N=121. At N=49 the DINOv2 similarity of 0.949 matches Gen3C's 0.949 exactly; the other three metrics are higher.
- Smaller backbone, stronger results: The authors note PPMem uses a Wan-2.2 5B backbone, smaller than Gen3C (Cosmos 7B) and Voyager (HunyuanVideo ~13B).
- Long-context stability: On full DL3DV test sequences (~350 frames in the plot), PPMem keeps the highest CLIP and DINOv2 similarity throughout, and the gap widens with frame index — leading the next-best method by ~0.27 on DINOv2 at frame 300. VMem collapses early; Gen3C and SPMem degrade steadily.
- Memory compactness: A merged pool holds ~100K primitives (~5 MB) for a full scene after ~400 frames, versus ~250K surfels, ~1–2M TSDF-derived points, and a ~210M theoretical total for accumulated per-pixel dense point clouds (counts estimated at ~400 frames, 576×1024).
- Representation ablation (Table 3): The joint plane pool (δ<1.25 = 70.3, AbsRel 0.210) beats no persistent memory (64.0, 0.272), a Gen3C-style MoGe-2 point cloud (55.0, 0.339), and post-hoc standalone Plana3R planes (57.0, 0.311); video PSNR is 14.91 versus 14.58 / 14.37 / 14.22.
- Depth conditioning ablation (Table 4): Removing depth conditioning collapses geometry (AbsRel 0.436, δ<1.25 = 30.7%) and lowers PSNR to 14.19. Channel concatenation alone (14.88, δ 72.3) and concat + DTF (14.91, δ 70.3) are comparable on the well-aligned ablation benchmark; the authors argue DTF's advantage appears with imperfect inference-time conditioning.
- Scale faithfulness probe: When input depth is rescaled by a known factor, concat + DTF tracks it at 1.00 ± 0.01 at scale 1.0, while concatenation alone responds in a compressed way at extreme scales. Natural-log RMSE to the identity mapping is 0.273 for DTF versus 0.320 for concatenation alone, and 0.131 versus 0.293 when both are trained with depth scale augmentation (s ~ logU[0.25, 4]).
- Decoder and loss ablation (Table 5): A VAE-initialized feature decoder improves δ<1.25 by +11.1 pp (59.2 → 70.3) and AbsRel by −0.046 (0.256 → 0.210) over the Plana3R upsampling head. Removing the rasterization loss costs −3.9 pp δ<1.25 and +1.2° normal error; removing the normal loss makes depth collapse to all-zero output.
Methodology in Plain English
The system generates video in chunks of about 25 frames. For each chunk, the diffusion transformer is given the noisy video latent plus conditioning: projected RGB from earlier views, visibility masks, and a partial depth map rendered from the memory of everything generated so far. The camera is communicated entirely through these projected views rather than explicit intrinsics, which keeps the model agnostic to specific camera parameters.
Alongside the video tokens, the transformer produces geometry tokens. A feature decoder (initialized from the video VAE's upsampling block) upsamples them 2× spatially, and three small MLPs regress each plane's 7 numbers: a center depth, two rectangular extents, and a rotation quaternion whose third column is the surface normal. Planes are rendered with differentiable rectangular Gaussian rasterization, giving depth and normal maps that can be compared to ground truth.
New planes are merged into a global pool using an observation-quality score s = r/|cos θ| (r is distance to the creating camera, θ the angle between viewing ray and normal; lower is better). Three rules keep the pool clean: occlusion deletion removes pool planes that newer observations show floating in front of real surfaces (via multi-camera voting); bidirectional quality dedup keeps only the better-scored plane among nearby ones, within a chunk (typically cutting count by ~80%) and across chunks; and a forward-looking prune drops planes that the next chunk will see from much closer. This merging is local and needs no gradient-based optimization.
The DTF module handles the fact that rendered depth has holes and partially covered patches. Depth is patchified at 16×16 and encoded by a ViT-Base (~87M parameters, ~1.7% of the 5B DiT, initialized from DINOv2-Base), then fused with DiT feature tokens. Invalid patches are always dropped, mixed patches are dropped with probability 0.75, and valid patches are randomly dropped to hit a target mask ratio in [0.6, 0.9], so the model tolerates sparse conditioning.
Training is two-stage. Stage 1 (~7,000 iterations) fine-tunes Wan-2.2 for camera-controlled generation with flow-matching loss. Stage 2 (~5,000 iterations) freezes the backbone and trains the plane head, DTF, and feature decoder with a geometry loss combining per-patch depth (log-space L1) and normal terms, rasterization depth/normal terms enabled after 1,000 warmup iterations, and a total-variation smoothness term on rendered depth; weights are λ_d=0.2, λ_n=0.05, λ_d^r=0.02, λ_n^r=0.01, λ_s=0.02. LoRA adapters (rank 256, α=256) are injected on self-attention in all 30 DiT blocks, making ~377M of 5.4B parameters (7.0%) trainable. At inference, LoRA is disabled for all denoising steps and enabled for a single extra forward pass that extracts planes, decoupling video quality from geometry.
Training data is DL3DV plus RealEstate10K (80K+ clips) at 576×1024, with metric depth from MoGe-2 scale-aligned to VGGT reconstructions. Crucially, training conditioning is produced by the same plane-based rendering pipeline used at inference — an offline incremental plane reconstruction over 24-frame segments with a one-step look-ahead — so the model trains on realistic holes and planar approximation errors rather than perfect projections.
Why This Matters
The paper reframes long-horizon video generation as a memory-representation problem: instead of scaling attention or retrieving frames, it argues for a structured, editable 3D state that is orders of magnitude smaller than per-pixel alternatives and is produced by the generator itself rather than a bolted-on depth estimator. If plane primitives suffice as memory, long video generation becomes cheaper to condition and the outputs become directly usable as 3D assets.
Real-world applications:
- Interactive walkthroughs of homes and buildings from a single photo, where exported wall/floor planes feed downstream floorplan or measurement tools (the RealEstate10K evaluation points this direction).
- Scene editing and relighting for film and content production, since geometry is explicit and exportable rather than locked inside a neural network.
- Simulation and robotics training environments, where metric-scale depth and normals support navigation and collision reasoning.
- Digital twins of indoor and urban spaces, given the emphasis on walls, floors, ceilings, and facades.
Industry relevance: the compactness (~5 MB per scene) and exportability of planar geometry matter for deployment on memory-constrained hardware and for interoperability with existing 3D content pipelines, while the frozen-backbone, LoRA-based training recipe lowers the cost of adding geometry to an existing video model.
Future Directions
- Joint end-to-end optimization. The two-stage design deliberately freezes the video backbone, preserving generation quality but forgoing co-adaptation of video and geometry objectives; the authors flag joint optimization as a more general alternative.
- Dynamic plane primitives. Extending the memory from static surfaces to moving planes is listed as a natural next step.
- Open-world scaling. Scaling beyond the indoor/outdoor real-estate and DL3DV domains is left open.
- Handling non-planar content. The representation is best suited to structured scenes; dense vegetation and water may be captured less faithfully than by point-based alternatives, and extreme camera trajectories unseen in training can degrade quality — a quantitative breakdown versus scene planarity is deferred to the appendix.
Target Audience
Researchers and engineers working on video diffusion, world models, and long-context video generation; 3D vision practitioners interested in compact scene representations and feed-forward plane reconstruction; and technical teams building scene-scale generative tools for real estate, simulation, or content production. Readers without a background in diffusion transformers and multi-view geometry will find the architectural and training details demanding, though the core memory concept is accessible.
Authors’ abstract
Video world models that generate and explore environments at scale need a persistent 3D memory of what they have produced. Plane primitives offer a natural candidate: a compact set of oriented planes captures the dominant structural elements (e.g., walls, floors, facades) of indoor and outdoor scenes, while requiring only 7 parameters per primitive. We present PPMem, a video world model that adopts plane primitives as its persistent 3D memory. PPMem augments a video…
The full abstract on arXiv