Memory as Plans: World-Action Modeling with Memory-Grounded Planning

AI summary

Overview

• Research area: Robotics — long-horizon memory-dependent robotic manipulation, vision-language-action (VLA) policies, and world-action models (WAMs).
• Technical level: Advanced. The paper assumes familiarity with transformer-based policies, flow matching, diffusion/DiT video backbones, KV caching, and mixture-of-experts style architectures.
• Scope: A framework that reinterprets long-term robot memory as compact, planning-time "plans" rather than a dense observation history fed to the action executor, validated on a memory benchmark and real hardware.

What This Paper Is About

Most robot policies assume the current camera view (or a short recent window) contains everything needed to act. But many real manipulation tasks are non-Markovian: the information required for the next decision was visible earlier and is now hidden. MaP-WAM addresses this by storing history as a compact record of completed subtasks and turning that record into a plan — a next-step language instruction plus a predicted visual trajectory — which a fixed-context executor then follows.

Key Contributions

1. Memory-as-Plans framework (MaP-WAM): Long-horizon episodic evidence is converted into memory-grounded plans, so the executor never has to condition on a growing observation history. Context length stays constant while fine-grained visual grounding is preserved.
2. Memory-grounded language-visual planning: A vision-language planner predicts the next segment-level language instruction, and a causal world model (CWM) generates a corresponding visual plan, translating prior visual evidence into concrete execution guidance.
3. Progress-aware execution: The World-Action-Progress (WAP) model jointly predicts future visual dynamics, action chunks, and normalized execution progress in a Mixture-of-Transformers design, enabling variable-duration plan execution, explicit temporal state disambiguation, and adaptive segment transitions.
4. Plan-observation alignment: At deployment, the current observation is matched against nearby frames of the predicted visual plan to calibrate the recursive progress estimate, mitigating drift on long tasks.
5. Strong empirical results with flat latency: 83.3% average success on RMBench and 78.0% on real-robot tasks, with roughly constant per-chunk executor latency (~827 ms) as history grows — versus a full-context baseline that exceeds 80 GB GPU memory at 1,700 frames.

Main Findings

• State-of-the-art on RMBench: MaP-WAM reaches 83.3% average success across nine tasks, beating LingBot-VA (77.1%) and far exceeding Memory-grounded baselines such as Mem-0 (42.0%), DP (5.8%), π0.5 (10.4%), and X-VLA (9.8%).
• Large gains on M(n) memory tasks: On tasks requiring multiple past observations, MaP-WAM hits 100% on Cover Blocks, 96% on Press Button, and 94% on Blocks Ranking Try, where most baselines struggle.
• Biggest improvement where memory is hardest: On "Observe and Pick Up," which requires distinguishing dozens of objects from temporally separated observations with only 50 demonstrations, success rises from 9% (best baseline) to 19%.
• Real-robot validation: On a 7-DoF Franka Research 3, MaP-WAM achieves 88% on Find Button and 68% on the harder Press Buttons task, where baselines fail entirely (non-zero baseline success on Find Button only comes from random selection).
• Visual memory matters, not just visual plans: Ablations show that removing the visual plan or replacing memory-grounded planning with a current-observation-conditioned world model both reduce success, indicating the visual guidance must be anchored in long-horizon evidence.
• Progress modeling is the single biggest ablation factor: Replacing progress with a binary completion classifier drops the average to 37.0%; removing the progress condition gives 54.0%; disabling plan-observation calibration gives 73.7% versus 96.7% for the full model.
• Progress calibration matters most on long horizons: On Blocks Ranking Try (average 423 steps per subgoal versus 116 for other tasks), recursive progress drift is significant, and calibration recovers most of the loss.
• Flat inference latency: CWM planning remains efficient past 16 completed segments thanks to block-causal masking and KV caching; the executor's cached plan prefix keeps per-chunk latency constant, while a full-context variant incurs ~4× zero-history latency at 1,500 frames.

Methodology in Plain English

The system splits memory handling from action generation into two stages.

Stage one — planning. Every time a subtask finishes, the robot records what it did: the language instruction for that segment plus a small set of frames (8 uniformly sampled stills) from the real execution. This forms an episodic memory. To plan the next subtask, a fine-tuned vision-language model looks at a handful of keyframes from each past segment and predicts a short language instruction for what comes next. A causal world model (built on a video diffusion transformer) then takes the full sparse visual history plus that new instruction and generates a short predicted video — a visual "storyboard" of what the next segment should look like.

Stage two — execution. The storyboard and instruction are handed to the World-Action-Progress model, which extends a pretrained video model with separate "experts" for actions and progress. It predicts the next chunk of actions, the corresponding future visual states, and a normalized progress value between 0 and 1 indicating how far through the plan the robot is. Because the plan is a fixed, static prefix, its internal key-value states are computed once and cached for the whole segment — the executor's input length never grows.

Closing the loop. Progress can't be measured directly at deployment, so it's predicted recursively, which drifts over long tasks. To correct this, each frame of the visual plan is tagged with a progress index; the system finds the plan frame most visually similar to what the robot now sees and nudges the progress estimate toward that frame's index. When the averaged predicted progress crosses a threshold (τ = 0.95), the segment is considered complete: real observations are resampled into new memory, and planning fires again for the next segment.

Training. Both planners and the executor use conditional flow matching. The executor's loss combines visual dynamics, action, and progress terms with equal weights. The progress condition is randomly perturbed during training to make the model robust to noisy estimates.

Why This Matters

Research impact. The paper challenges a common assumption in embodied AI: that long-horizon memory must be continuously fed to the action model. It argues that history is needed primarily at planning time, and that decoupling the two preserves both accuracy and efficiency. This framing could reshape how future VLA and world-action architectures handle memory, and it introduces progress as a first-class generated modality rather than an after-the-fact verifier.

Real-world applications:

• Household and service robots that must remember where an object was placed earlier (e.g., which drawer a tool went into) before it was occluded.
• Warehouse and logistics picking, where an item observed on a shelf must be retrieved after the robot's view has moved on.
• Assembly and inspection, where a sequence of steps must be executed over hundreds of control steps with visually similar intermediate states that need disambiguation.
• Assistive robotics, where a user's earlier instruction or demonstration is no longer visible but determines what the robot should do next.

Industry relevance. The efficiency result is practically significant: a full-context executor runs out of GPU memory at 1,700 frames, while MaP-WAM holds steady latency. For deployed robots where action generation dominates the compute budget, keeping per-chunk latency constant as tasks get longer is what makes long-horizon autonomy economically and physically feasible.

Future Directions

• Automatic segment discovery. MaP-WAM currently relies on segment boundaries provided by the benchmark's annotations. Extending it to unsegmented demonstrations — discovering natural subtask boundaries from raw trajectories — is the most direct next step.
• Learned plan-observation similarity. The alignment step uses a lightweight, training-free matching metric. A learned similarity measure could improve calibration in visually cluttered or ambiguous scenes.
• Longer and more compositional memory. The current episodic record is flat (a list of completed segments). Hierarchical memory that summarizes groups of segments could scale to tasks with dozens or hundreds of subtasks.
• Failure recovery and replanning. The framework triggers replanning only on successful segment completion. Detecting and recovering from stalled or failed progress predictions — and replanning mid-segment — remains open.

Target Audience

• Robotics and embodied AI researchers working on VLA policies, world models, and long-horizon manipulation.
• ML engineers building robot learning systems who need memory mechanisms that do not blow up latency or GPU memory.
• Graduate students studying memory architectures, plan-conditioned policies, or flow-matching-based action generation.
• Industry practitioners in warehouse automation, service robotics, and manufacturing evaluating whether memory-aware policies are deployable at realistic compute budgets.

Authors’ abstract

Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation. Existing memory mechanisms often rely on language summaries, growing visual windows, or their combinations, and may therefore lose fine-grained visual evidence or face a trade-off between history coverage and execution efficiency. We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history. MaP-WAM represents memory as completed segment records containing language instructions and sparse visual context, and converts this episodic memory into compact plans comprising the next segment-level language plan and corresponding visual guidance. A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predicting action chunks and corresponding execution progress at inference time, calibrating predicted progress through plan-observation alignment for adaptive segment transitions and closed-loop context updates. MaP-WAM keeps the executor context length fixed, while structured attention further enables key-value caching in both planning and execution. MaP-WAM achieves state-of-the-art performance on RMBench with an 83.3% success rate and attains 78.0% success on real-robot tasks, while maintaining approximately constant executor inference latency as task history grows.