WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

AI summary

Overview

Research area: Computer vision — evaluation methodology for generative video "world models" (camera-controlled, autoregressive video generators that simulate explorable environments).

Technical level: Intermediate. Readers should be comfortable with video diffusion/autoregressive generation, camera pose representations, and standard perceptual metrics (DINOv2, ArcFace, LPIPS-style comparisons); the benchmark protocol itself is explained from first principles.

Scope (one sentence): The paper introduces WorldMemBench, a 260-case benchmark that measures whether video world models can remember static content (humans, objects, environments) and correctly evolve dynamic events across camera departures and repeated revisits, evaluated over nine recent models.

What This Paper Is About

Video world models now generate long, interactive rollouts where the camera repeatedly looks away from and back at content. A good-looking video is not enough: the world must remember what was already seen and must keep events progressing sensibly while they are off-screen. The paper builds a systematic testbed that separates these two abilities — static memory and dynamic memory — and uses controlled "departure and revisit" camera paths to check whether a model's world stays consistent with its own history.

Key Contributions

1. A 260-case benchmark across 23 categories. WorldMemBench contains 200 static-memory cases (60 Human, 70 Object, 70 Environment) and 60 dynamic-memory cases covering seven types of off-screen state transitions, evaluated along four dimensions: Static Memory, Dynamic Memory, General Video Quality, and Camera Following.

2. A fixed-reference multi-revisit protocol for static memory. Each return is compared against the same initial reference, with same-axis returns (LRLR, LRLRLR) and cross-axis returns (LRUD, a yaw cycle followed by a pitch cycle) to expose degradation beyond a single revisit. Dynamic cases instead specify an initial state, an action, and an expected next state, scored with event-specific checklists that separate event initiation from event continuation.

3. Human-validated automatic metrics. Blinded A/B agreement between automatic static metrics and human judgments ranges from 78.5% to 92.5% across six metrics, with model-level Spearman correlations from 0.301 (Environment Depth) to 0.988 (Environment Visible); the Dynamic Memory Overall score reaches Spearman ρ = 0.732 and PLCC = 0.808 over 50 instances.

4. A nine-model evaluation plus a controlled context-access case study. The benchmark is applied to HY-WorldPlay, Matrix-Game 3, Matrix-Game 3.5, SANA-WM, YUME, LingBot-World, LingBot-World v2, Evoke, and DreamX-World, and then reused to compare three training-free context policies (FullKV, Sliding Windows, FoV) on LingBot World v2 while holding the model fixed.

Main Findings

• No model leads on all four dimensions. HY-WorldPlay is best on all seven Static Memory metrics (Human ID 0.828, Human Appearance 0.830, Object Appearance 0.879, Object Geometry 0.761, Environment Visible 0.947, Environment Depth 0.905, Environment Reconstruction 0.619), yet its Dynamic Memory Overall is only 0.294. LingBot-World leads Dynamic Memory Overall at 0.808, with YUME second at 0.775.

• Static retention does not imply dynamic state evolution. HY-WorldPlay initiates only 31.7% of prescribed events, so its dynamic scores collapse despite near-saturated static scores. The paper presents this as evidence that preserving appearance and simulating event progression are distinct capabilities.

• Camera control and memory quality are separable diagnostics. Matrix-Game 3 records the lowest translation RPE (0.424) and HY-WorldPlay the lowest rotation RPE (0.417), while other models such as SANA-WM and DreamX-World show much larger errors (translation 1.018 and 1.139). Reporting these separately lets readers tell memory failures apart from viewpoint failures.

• Memory decays across successive revisits. Across nine-model averages, all eight tracked metrics degrade monotonically from the first to the third revisit under LRLRLR: Object Appearance drops by 0.229, Human Identity by 0.217, Object Geometry by 0.205, and Environment Reconstruction by 0.196. Per model, every model is weaker at R3 than R1 on Human, Object, Environment Visible, and Environment Reconstruction, and Environment Depth degrades for eight of nine models.

• Decay persists even when the return viewpoint is correct. Among 431 Environment model–case pairs that satisfy the departure and return criteria, the nine-model mean Visible, Depth, and Reconstruction scores still fall by 0.152, 0.135, and 0.139 from R1 to R3. Mean Camera Rotation RPE also grows from 0.430 to 0.502 to 0.530 across the three revisits.

• Switching exploration axes is harder than repeating one. Comparing the second revisit of LRUD against LRLR, mean Δ_axis is positive for all seven static metrics, and 50 of 63 model–metric pairs degrade more under LRUD; Environment Visible shows the largest additional drop, at 0.152.

• Context selection trades memory dimensions against each other. On LingBot World v2, the FoV policy (recent context plus view-relevant history) is best on eight of eleven capability metrics and beats Sliding Windows on all eleven, and it beats FullKV on all four Dynamic Memory sub-metrics and both Human metrics. FullKV retains an advantage on Object Appearance (0.564 vs 0.531), Object Geometry (0.506 vs 0.485), and Environment Visible (0.694 vs 0.687). Across 660 matched videos, end-to-end latency is 322.2 s for FullKV versus 262.7 s for FoV and 260.4 s for Sliding Windows (1.23× and 1.24× speedups).

Methodology in Plain English

The researchers build each test case around a controlled camera script. A model first sees a subject or scene, the camera then moves or turns away, and later comes back. Because the return is prescribed, every revisit can be compared to the same initial frames rather than to a drifting reference, so error accumulates visibly instead of being averaged away.

Two kinds of motion are used. Translation keeps the camera's facing direction fixed and walks it Forward–Right–Backward–Left in a square loop. Rotation keeps the camera in place and pans; LRLR and LRLRLR repeat a left–right cycle two and three times, while LRUD does a left–right cycle and then an up–down cycle so that the second return arrives from a different axis.

For static memory, each target is specified by a mask in the conditioning frame and then re-found at each revisit. Human identity and appearance are compared with ArcFace embeddings and DINOv2 features inside person masks. Object appearance uses DINOv2 inside object masks, and object geometry compares silhouette overlap (IoU) after each mask is cropped to its bounding box, resized with aspect ratio preserved, and centered on a common canvas. Environment memory is split into Visible (DINOv2 similarity to the frame whose estimated camera orientation is closest to the initial one), Depth (rank correlation of depth order, which ignores absolute scale), and a Spherical Reconstruction Score that projects depth onto a shared spherical grid after removing global depth scale, penalizing missing surfaces, extra surfaces, and distance errors.

For dynamic memory, each case states an initial state, an action, and an expected next state. A judge model (Qwen3.8-27B) first decides whether the target event visibly starts; if it does not, all eight checklist answers score zero. Otherwise it answers eight binary questions — two each on return progress, persistence/reset, final state, and causal/physical consistency — against fixed reference answers. This gating prevents a model from scoring well on an event that never happened.

Video quality is measured with MUSIQ-SPAQ (imaging) and the LAION aesthetic predictor (aesthetic) on every fourth frame. Camera following uses translation and rotation relative pose error, estimated with DA3 and ViPE and aligned with Sim(3) for translation, with rotation normalized by the target angular path. All rollouts were produced on machines with eight NVIDIA H20-3e GPUs using frozen seeds and each model's released default configuration.

Why This Matters

For research, this paper converts "memory" from an informal claim into a set of separately scored, human-validated measurements, and shows that the field's implicit assumption — that better visual fidelity or better static consistency implies a better world model — does not hold. The multi-revisit and cross-axis protocols also reveal failures that single-return evaluations structurally cannot see, which matters for anyone reporting progress on long-horizon interactive generation.

Real-world applications that depend on persistent generated worlds:

• Interactive games and simulation, where a player walking away from a location and back must find it unchanged, and where off-screen events must have consequences.
• Embodied AI and robot policy training, where generated environments serve as training grounds and must obey consistent physics and object permanence across episodes.
• Virtual production, digital humans, and telepresence, where a person's identity and appearance must survive repeated camera motion.
• World-model-based planning, where an agent's imagined rollouts are only useful if the imagined state evolves causally rather than resetting.

For industry, the benchmark has two immediate uses: model selection, since the nine-model table shows different models winning different dimensions, and inference engineering, since the context-access study shows a retrieval policy that improves most memory metrics while cutting latency from 322.2 s to 262.7 s on matched videos. Producing long rollouts is expensive, so a 1.23× speedup with better dynamic memory is a directly actionable trade-off.

Future Directions

• Diagnosing and fixing cumulative decay. The paper attributes degradation to compounding viewpoint misalignment plus autoregressive error propagation but does not propose a remedy; methods that re-anchor to earlier observations or correct drift across revisits are the obvious next step.
• Cross-axis robustness as a design target. Because 50 of 63 model–metric pairs degrade more when the return axis changes, it is open whether this stems from training data biased toward yaw motion or from the pose conditioning interface itself.
• Better context-access policies. FoV beats FullKV on dynamic and human metrics but loses on object appearance, geometry, and environment visibility, so the question of how to allocate a context budget per memory dimension remains unresolved.
• Extending scope and scale. The benchmark covers 260 cases and rollouts of roughly 20–35 seconds; whether these findings hold for much longer horizons, other control interfaces (action- or language-driven), and broader content categories is untested.

Target Audience

Researchers and engineers building or evaluating video world models and long-horizon video generators; benchmark designers who need a template for fixed-reference, multi-revisit evaluation protocols; and practitioners choosing among camera-controlled video models for interactive applications. Readers interested in evaluation methodology more broadly will also find the human-validation protocol and the separation of diagnostics (video quality, camera following) from the target capability (memory) useful as a model for benchmark design.

Authors’ abstract

As video models enable longer and more realistic interactive experiences, maintaining a world consistent with its history becomes essential. This requires both preserving previously observed content and ensuring coherent world evolution over time. World memory is a critical capability to evaluate alongside visual fidelity and video duration. We introduce WorldMemBench, a benchmark for systematically evaluating world memory in video world models. Across 260 test cases, it assesses static memory through the retention of human identity and appearance, object appearance and geometry, and environmental content and geometry, and dynamic memory through event progression and state persistence. Using controlled camera departures and multi-revisit trajectories, the benchmark evaluates static content against fixed initial references and dynamic events against expected state transitions. It combines appearance and geometry metrics with event-specific assessments, and reports video quality and camera-following accuracy as complementary diagnostics. Evaluation of nine representative models shows that strong static retention does not imply reliable dynamic state evolution, while repeated revisits reveal memory degradation beyond the first return. Human studies support the alignment between our automatic evaluations and human judgments. WorldMemBench provides a systematic testbed for diagnosing memory failures and guiding the development of persistent, temporally coherent video worlds.