Execution Before Memory: Diagnosing Memory Failures in Video World Models

AI summary

Overview

Research area: Computer vision — evaluation of generative video "world models," specifically how memory of hidden scene state is measured. Technical level: Intermediate (assumes familiarity with video diffusion models, VLM judges, and benchmark scoring). Scope: The paper introduces MIST, a staged evaluation that separates whether a scene is set from whether an instructed change is actually executed before memory is scored, plus M3-Agent, a test-time instruction-revision loop demonstrated on Wan TI2V-5B, Wan T2V-A14B, and a MiniMax-H3 pilot.

What This Paper Is About

Video world models are scored on whether they "remember" a scene after the camera looks away, but existing benchmarks collapse scene setup, the instructed change, and the final frame into a single number. If the model never carries out the change (never opens the box, never sits in the chair), the hidden state transition never appears, so a low score gets misread as forgetting when memory was never actually tested. The paper builds an evaluation that scores these stages separately, shows the same failure inside official benchmark clips, and proposes a test-time method that revises the failed instruction so a frozen generator actually performs the missing action.

Key Contributions

1. MIST, an evaluation that scores scene setup, leakage of the hidden answer during the gap, and the final reveal separately, using three 5-second shots per case with a fixed VLM judge, so a missing change is not reported as forgetting.
2. Evidence that official benchmark failures are execution failures: failed WRBench clips skip the departure or return, and failed MBench-T clips return to an object that is still intact — the same classes MIST isolates.
3. M3-Agent (Make the Missing Move), a test-time workflow that reads the last frame, names the failed operation, rewrites only that operation in the instruction, and regenerates with the same frozen model — no memory module added or trained.
4. Ablations separating instruction types: look-then-rewrite versus answer-only, same-sentence resampling, open-and-hold, and rewrite-without-answer, plus text-only and scorer-free controls that bound what the gains can mean.

Main Findings

• The bottleneck is execution, not memory. In the MIST setting on Wan TI2V-5B (three 5-second shots, 43 cases), setup scores 0.852 while baseline reveal scores 0.147; 41 of 43 baseline clips never open. Setup is at least 0.8 while baseline reveal stays below 0.3 in 38 of 43 cases.
• The hidden answer does not leak. Paired gap leak is 0/43 for both the baseline and the later rewrite; a broader 51-row baseline export also contains no leak.
• Revising the failed move changes the reveal gate, not the setup. With one kept clip per case, reveal rises from 0.147 to 0.492 and inside match from 0.047 to 0.442 while setup moves from 0.852 to 0.851 (−0.001). Never-opens fall from 41 to 20; correct clips rise from 1 to 13; open-but-wrong rises from 0 to 10.
• The rewrite is not uniformly safe. Per case, reveal rises in 35 of 43, stays the same in 6, and falls in 2; mean paired change is 0.34 (95% CI 0.24 to 0.45). Case R2-10 goes from correct to never-open, which is why the original clip stays eligible.
• All three operation families move the same way. Reveal goes from 0.132 to 0.549 on presence and count, from 0.147 to 0.464 on binding, and from 0.163 to 0.454 on history update, with setup flat in each.
• Rewriting beats naming the target, but only when the target is named. With five tries per arm, look-then-rewrite reaches 0.763 reveal and 0.721 inside versus 0.684 and 0.581 for answer-only and 0.635 and 0.512 for the same sentence resampled. The same rewrite with the target withheld falls to 0.174 reveal with 28 of 43 lids still closed; bare "open and hold" reaches 0.268.
• WRBench: T rises 26.1% (0.452 to 0.570, n=40). The gain comes from the camera finishing the departure and return, not from better-looking frames — a component audit attributes about 3–4% of ΔT to visual quality, 55–56% to the departure check D5′, and 41% to the return check D6′. Of the 40 return gates, 18 stay open, 13 newly open, none close, and 9 stay shut. The paired gain has a 95% bootstrap interval of +0.070 to +0.166.
• MBench-T: U rises 52.7% (0.292 to 0.446, n=30), but selection does the work. Selection keeps the baseline on 23 of 30 items, the leave–return instruction on 5, and the last-frame lock on 2. Forcing leave–return as a fixed instruction drops U to 0.251 and harms 8 cases whose baseline state score is 1.0; a forced last-frame lock drops U to 0.228. One clip per item gives 0.443.
• Search is not monotone. On WRBench, adding a second instruction set lowers T from 0.550 to 0.547 before the last step restores it to 0.570.
• Text can name the target without showing it. A held-out 8B reader names the gold target from text alone at 0.91, drops to 0.40 on a setup caption and 0.07 on shuffled gold, but accepts almost no baseline last frame (0.09, and 0.023 for the 122B judge), rising to 0.56 (8B) and 0.442 (122B) after the rewrite.
• Scorer-free selectors do not beat the baseline. First rewrite only, 20 random draws, and a blind 8B last-frame reader all produce intervals overlapping the baseline on both suites (for example WRBench 0.466 for the blind reader versus 0.452 baseline), whereas keep-best reaches 0.570.
• Packed longer takes change the failure mode. On a packed 15-second MiniMax-H3 take the same judge finds 3 of 43 never-open and 24 of 43 opened onto the wrong state — the failure moves from not opening to opening wrong.
• Ceilings sit above the reported method. On WRBench, a peek at the gate followed by visual selection reaches 0.586 and the maximum T in the pack is 0.601, versus the reported 0.570.

Methodology in Plain English

The researchers rebuilt the memory test so that each stage can be scored on its own. Each case is three 5-second shots: the first establishes the scene, the second passes through an opaque gap where the answer must stay hidden, and the third asks the model to reveal the changed state. All reveal comparisons reuse the same locked first two shots, so only the final shot varies. A fixed VLM judge then checks four things — whether the setup is right, whether the answer leaked into the gap, whether the required action actually happened, and whether the visible final state matches the target. These are reported as separate numbers rather than multiplied into one total.

For the instruction experiments, the team inspected the last frame, labelled the specific failure (no action, wrong binding, missing departure, return to an intact object, occluded view), and rewrote only the failed operation plus the clause describing what must be visible at the end — never the actor, object, target state, or temporal order. Five instruction strategies were compared under the same five-try budget: look-then-rewrite, answer-only, same-sentence resampling, open-and-hold, and rewrite with the target withheld (audited at 0/43 leaks). The same diagnose-and-rewrite loop was then applied to official WRBench and MBench-T protocols, keeping their own scorers and camera clauses, with the original clip always remaining a candidate so a harmful rewrite can be refused. Everything runs on frozen generators; no module is trained.

Why This Matters

Evaluation practice: the paper argues that a memory benchmark should publish its checks separately rather than as one total, because a single number cannot distinguish a missing action from a wrong state from genuine forgetting. It also shows that headline gains from test-time search are oracle coverage estimates when the selector uses the same scorer that reports the result — a caveat that applies well beyond this paper.

Real-world applications (implied by the subject matter, not reported as results in the paper):

• Robotics and embodied simulation, where a planner needs to know whether a simulated world actually executed an action before trusting what it "remembers" about the outcome.
• Long-horizon video generation and editing tools, where a closed lid or unchanged object silently breaks a shot and a targeted instruction fix is cheaper than reshooting.
• Quality assurance for generative media pipelines, where separating setup, action, and final-state errors tells a team which stage to fix.
• Benchmark and leaderboard design, where vendors and reviewers need scores that cannot be gamed by text priors or by resampling the same prompt.

Industry relevance: the method requires no retraining and no memory module, so it is directly applicable to deployed frozen generators; at the same time, the paper is explicit that keep-best selection is not a deployable critic, which limits how the headline numbers can be used in product claims.

Future Directions

• A target-blind, deployable selector. The paper states this explicitly: claiming the backbone stored the target state would require a selector that beats the baseline without seeing the scorer, held-out cases, and human labels on the exposed state.
• Separating scale from protocol. The authors note that the TI2V-5B versus T2V-A14B settings differ jointly (clip length, prompting, seeds), so model scale is not a controlled comparison here.
• Broader generator and stitch coverage. The main study uses one generator family under Internal and Official protocols, and the packed MiniMax-H3 cells remain Internal and qualitative; the H3-43 122B reveal moves from 0.387 to 0.729 with 3 of 43 never-open, but the paper does not claim this transfers to the official scores.
• Rewrites that never damage a correct case. Because R2-10 flips from correct to never-open and forced MBench-T interventions hurt 8 cases, making revision monotone — or reliably refusing harmful edits without the reporting scorer — is an open problem.

Target Audience

Researchers and practitioners working on video generation, world models, and embodied simulation who need to evaluate what a model actually retains; benchmark designers who want stage-resolved metrics instead of a single aggregate; and engineers deploying frozen video generators who want test-time fixes that do not require training. Readers should be comfortable with diffusion video models and VLM-as-judge evaluation, though the central argument — check whether the action happened before calling it forgetting — is stated in plain language.

Authors’ abstract

A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory. We introduce MIST, an evaluation that separates the change from the memory it should leave behind. The model first sets the scene while the change stays hidden, and only later is asked to show it, with leakage before that point kept under control. This separation locates the bottleneck: the model sets the scene but does not carry out the instructed change. The same gap appears in general memory evaluations. On that basis we propose M3-Agent (Make the Missing Move), which recursively revises the details of the input instruction so that the model successfully carries out the requested instruction. Relative to the baseline, it more than triples the MIST reveal score. On the benchmarks, WRBench's score T rises by 26.1% and MBench-T's score U by 52.7%. The method meets the problem from the execution side: it shows what the frozen model can do once the move is stated, and it sets a stricter baseline.