R2M-Bench: Evaluating Revisit Memory via Relative Consistency in Interactive Video World Models

AI summary

Overview

Research area: Computer vision / generative video world models — specifically, benchmarking whether action-conditioned video world models remember a scene after the agent leaves and returns.

Technical level: Advanced. The paper assumes familiarity with diffusion/autoregressive video generation, chunked rollout inference, retrieval-augmented conditioning, visual place recognition descriptors, and two-view geometry (epipolar constraints, RANSAC).

Scope in one sentence: The paper introduces R2M-Bench, a 300-instance benchmark that scores revisit memory as a relative advantage over within-video temporal controls rather than as absolute first-visit/return similarity.

What This Paper Is About

Interactive video world models generate long rollouts in response to camera or action commands, and they are supposed to keep a scene persistent when an agent leaves a room, street corner, or object arrangement and later comes back. The problem is that existing evaluation measures raw similarity between the first-visit frame and the return frame, and that score is easily inflated: a rollout that barely moves, renders conservatively, or repeats textures will have high first-visit/return similarity even with no real memory. This paper builds a benchmark that measures whether consistency is selectively elevated at the commanded revisit relative to ordinary, non-revisit moments sampled from the same generated video.

Key Contributions

1. R2M-Bench, a 300-instance benchmark. 100 diverse reference scenes are paired with each of three complementary leave-and-return trajectory templates (out-and-back, translation–rotation, closed-loop), and each instance is evaluated across five dimensions of visual and state consistency.

2. A same-video relative scoring protocol. Each detected revisit pair is compared against two controls drawn from the same rollout: a gap-matched non-revisit pair and a short-range pair. This yields MemoryGain (MG), the revisit advantage over the temporal baseline, and the Normalized Memory Ratio (NMR), which divides that advantage by the short-to-baseline dynamic range.

3. A multi-model empirical study with human validation and motion diagnostics. Seven action-conditioned video world model variants be evaluated, with human consistency judgments, an optical-flow-based motion diagnostic, controlled motion and quality diagnostics, and trajectory-wise profiling, characterizing both the alignment and remaining limitations of relative revisit scoring.

Main Findings

• NMR tracks human judgment moderately well. On 210 sampled rollouts (10 rollouts from every model–trajectory combination, 7 × 3 × 10), Overall NMR correlates with human consistency judgments at Spearman's ρ = 0.547, with a 95% bootstrap interval of [0.45, 0.63].

• Relative scoring reduces the slow-motion shortcut. In a within-model optical-flow diagnostic, NMR's correlation magnitude with generated motion is 0.072, compared with 0.207 for raw revisit similarity.

• DreamX-World-Memo leads the seven video models. It achieves the highest Overall NMR among the evaluated video models (0.706), followed by HY-WorldPlay (0.485), Matrix-Game 3.0 (0.403), and Lyra-2 (0.310). LingBot-World-2-Fast reaches 0.141, LingBot-World-Fast 0.100, and SANA-WM 0.016.

• The separate 3D reference sits well above all video models. WonderWorld, an explicit 3D reconstruction and rendering system reported as a near-perfect-memory reference rather than a comparable video model, reaches 1.510 Overall NMR.

• DreamX-World-Memo's lead holds on both reported readouts. It has the largest MemoryGain on 8 of the 11 displayed metrics and the largest NMR on 9 of 11.

• Aggregate rankings hide dimension-specific strengths. DreamX-World-Memo leads all three Appearance Fidelity metrics, two of three Scene Identity metrics, both Local Geometry metrics, and Persistent State. HY-WorldPlay is strongest on DINO scene similarity and ranks second overall, while Matrix-Game 3.0 has the highest semantic object-persistence score.

• Retrieval-based systems occupy the top four Overall NMR positions, but this is associative only. Because model scale, training data, generation procedure, and inference also differ, the paper states that this ranking does not isolate the effect of retrieval.

• Trajectory templates expose different failure modes. DreamX-World-Memo has the broadest profile on Out-and-back and Translation–rotation but contracts sharply on Closed-loop. HY-WorldPlay is less dominant on the first two templates yet retains a comparatively broader closed-loop profile, especially on scene and object cues. The two LingBot variants and SANA-WM show limited revisit-selective advantage once the return includes substantial viewpoint change or accumulated rollout error.

• MG and NMR play distinct roles. MG (in metric-specific units, e.g., PSNR, SSIM, LPIPS, cosine similarities) is treated as direct evidence of a revisit advantage, while NMR is used for cross-metric and family-level aggregation. NMR is not bounded by one; values above one indicate that the revisit advantage exceeds the corresponding short-to-baseline range and should be interpreted jointly with MG and the dynamic range.

• SANA-WM's MemoryGain results are near zero or slightly negative across the displayed metrics (for example DINO-sim −0.004, MVPR-sim −0.019, State Cons. −0.008), and its Overall NMR is 0.016.

Methodology in Plain English

Building the test cases. The researchers took 100 reference images of navigable scenes and wrote a text prompt for each. The primary scene partition is 24 indoor, 44 urban, 28 natural, and 4 abstract scenes; non-exclusive attributes mark 87 scenes as object-rich and 76 as outdoor. Each reference is paired with three model-agnostic navigation scripts built from simple primitives (forward–backward and left–right translation, left/right yaw). The same script is converted into whatever input format each model needs — discrete actions for action-only interfaces, camera deltas or poses for camera-conditioned ones — and integrated into a commanded pose sequence.

Generating rollouts. Each of seven action-conditioned video world models is run with chunked autoregressive inference, where past frames or model-specific memory states are fed back as context for the next chunk, on an 8-GPU NVIDIA H20 server. Videos contain approximately 481 frames with small model-specific padding variations; static leading and trailing frames are trimmed.

Mining three kinds of frame pairs. From the commanded poses, the benchmark mines revisit pairs: frame pairs far apart in time whose commanded positions and yaw angles return to the same region. For each revisit pair, it then samples a gap-matched baseline pair with a similar temporal gap from the same video, deliberately without a spatial-far constraint so that slow or near-static rollouts raise the baseline instead of being excluded. It also samples short-range pairs over a small temporal window to estimate the model's local consistency.

Scoring. All pairwise metrics are oriented so larger means more consistent. MemoryGain is the average revisit score minus the average baseline score. The dynamic range is the average short-range score minus the average baseline score. NMR is MemoryGain divided by (dynamic range + a small constant for numerical stability), and cases with dynamic range ≤ 10⁻⁸ are discarded. Overall NMR first averages metrics within each family, then averages the five families equally, so families with more metrics do not get extra weight.

What the five families measure. Appearance Fidelity uses PSNR, SSIM, and LPIPS for pixel-, structure-, and perceptual-level change. Scene Identity encodes frames with DINOv2 plus two place-recognition descriptors, BoQ and MutualVPR, and compares cosine similarity of global descriptors. Object Identity detects and segments anchors with GroundingDINO and SAM2 in the earlier frame, re-detects the same concepts independently in the paired frame, and scores masked DINOv2 appearance features and CLIP-based semantic compatibility, assigning zero to unmatched anchors. Local Geometry extracts SuperPoint keypoints, matches them with LightGlue, and tests the matches for a common two-view geometry with RANSAC, reporting a feature-correspondence ratio and an inlier ratio. Persistent State uses a vision-language model (Gemini-3.1-Pro-Preview, also used to draft prompts) with a fixed, pair-type-blind rubric that returns a scalar score for every revisit, baseline, and short-range pair.

Validation. Six annotators independently rated revisit consistency, generated motion, and camera control from 1 (Bad) to 3 (Good) on the 210 sampled rollouts, and their ratings were averaged per video.

Why This Matters

Impact on research. The paper reframes revisit-memory evaluation as a calibration problem: instead of asking "how similar are the first visit and the return?", it asks "how much more similar is the return than comparable non-revisit moments in the same rollout?" This makes scores less sensitive to rendering quality, repetitive content, and failed motion, and it gives the world-model community a shared protocol for a capability — persistence of a place after leaving it — that general video and world-model benchmarks had not cleanly isolated.

Real-world applications.

• Training and evaluating embodied agents and robot planners that rely on simulated environments to remember where objects and rooms are after leaving the field of view.
• Game and interactive-media world generation, where a player expects a location to look the same when they walk back into it.
• Autonomous driving and navigation simulation, where loop closure and viewpoint-changing revisits determine whether a synthetic scene is usable for testing.
• AR/VR scene generation and reconstruction pipelines, where a rendered location must remain consistent as the user turns away and returns.

Industry relevance. The benchmark evaluates seven action-conditioned video world model variants spanning 2.6B to 14B parameters — LingBot-World-Fast, LingBot-World-2-Fast, Matrix-Game 3.0, HY-WorldPlay, DreamX-World-Memo, SANA-WM, and Lyra-2 — from teams including several large industrial labs, and it reports an explicit 3D rendering reference (WonderWorld). Revisit memory is a direct gating capability for products that depend on long interactive rollouts, and the paper provides a measurement tool plus a public code repository (https://github.com/AMAP-ML/R2MBench).

Future Directions

• Separating retrieval mechanisms from confounds. The paper explicitly declines a causal conclusion about retrieval: the four retrieval-based systems rank top four on Overall NMR, but architecture, scale, training, data, generation procedure, and inference all differ. A controlled study varying only the memory mechanism would be a natural follow-up.

• Stabilizing NMR when the dynamic range is small. The authors note that NMR depends on both the revisit gain in the numerator and the dynamic range in the denominator, and can become unstable or amplified when the dynamic range is small; they report using MG as direct evidence and interpreting NMR jointly with MG and dynamic range. A more robust normalization is left open.

• Explaining why models fail mainly on closed-loop returns. Even the strongest systems degrade on longer closed-loop trajectories, and the paper reports that some models preserve Appearance Fidelity while Scene Identity or Local Geometry deteriorates. Diagnosing whether this stems from accumulated control error, autoregressive drift, or context attenuation is an open question.

• Extending beyond visual and semantic consistency. The five evaluated families cover observable consistency only. Whether relative revisit scoring transfers to other environment properties — physics, interactable object state, or long-horizon task outcomes — is not addressed.

Target Audience

Researchers and engineers working on video generation, interactive world models, embodied AI, and simulation environments. It is most useful to people who need to benchmark or build models with persistent scene memory, and to evaluation specialists interested in relative scoring designs that control for motion and rendering confounds. Readers need a working knowledge of video diffusion or autoregressive video generation and of standard vision metrics (PSNR, SSIM, LPIPS, keypoint matching) to follow the metric design; the motivation and the main ranking results are accessible to a broader technical audience.

Authors’ abstract

High similarity between first-visit and return frames does not necessarily show that a video world model remembered the scene; the intervening rollout may simply have changed very little. This ambiguity makes absolute revisit scores sensitive to rendering stability, repetitive content, and failed motion. We introduce \emph{R2M-Bench} (\textbf{R}elative \textbf{R}evisit \textbf{M}emory Benchmark), a benchmark of observable revisit-selective consistency. For every detected return, R2M-Bench compares the revisit pair with two controls from the same rollout: a gap-matched non-revisit pair that measures generic temporal stability and a short-range pair that estimates short-horizon consistency. These comparisons produce \emph{MemoryGain} (MG), the revisit advantage over the temporal baseline, and the \emph{Normalized Memory Ratio} (NMR), which normalizes this advantage by the short-to-baseline dynamic range. R2M-Bench combines 100 reference scenes with three leave-and-return trajectories to form 300 instances and evaluates appearance fidelity, scene and object identity, local geometry, and persistent state. Across seven action-conditioned video world models, Overall NMR correlates with human consistency judgments at Spearman's $ρ=0.547$ (95\% CI $[0.45,0.63]$). Its within-model correlation magnitude with generated motion is $0.072$, compared with $0.207$ for raw revisit similarity, indicating that relative calibration substantially reduces the slow-motion shortcut. DreamX-World-Memo achieves the highest Overall NMR among the evaluated video models. Together, these results support same-rollout relative calibration as a practical way to distinguish revisit-specific consistency from generic temporal stability.