HeLM: Hierarchical Explicit Language Memory for Non-Markovian Manipulation

AI summary

Overview

• Research area: Robotics / Vision-Language-Action (VLA) models, with a focus on memory for long-horizon and non-Markovian manipulation.
• Technical level: Intermediate. The core idea is intuitive, but the paper assumes familiarity with VLA architectures, policies, and imitation-learning evaluation protocols.
• Scope: The paper proposes HeLM, a hierarchical VLA framework that stores task-relevant history as explicit natural-language memory, and evaluates it on 10 real-world non-Markovian manipulation tasks (56 variants), including five newly defined inter-episode tasks.

What This Paper Is About

Most VLA robot policies assume the current camera image is enough to decide the next action (a Markovian assumption). Many real tasks break this assumption: after wiping part of a window or pressing a button twice, the scene looks the same as before, so a memoryless policy repeats or skips steps.

The paper's goal is to give a robot an explicit, inspectable record of what has already happened. It does this by keeping that record in plain language (a "memory" the model writes itself) and by only updating it when a meaningful event occurs or the instruction changes.

Key Contributions

1. HeLM framework: A hierarchical design in which a VLM-based high-level policy (HLP) maintains language memory and issues action commands, while a VLA-based low-level policy (LLP) executes them. Memory updates are event-triggered through factorized event detection rather than happening continuously.
2. Structured memory with two fields: Working memory tracks progress inside a single episode, and episodic memory carries summarized context across instruction boundaries, with working memory promoted into episodic memory at episode end.
3. A new task suite: 10 real-world non-Markovian tasks with 56 variants, split into intra-episode tasks (A1–A5) and inter-episode tasks (E1–E5). The inter-episode setting, where a later instruction depends on information from an earlier episode, is framed as an underexplored axis of robotic memory.
4. Component-level evidence: Controlled comparisons against a visual keyframe variant, a Perceptual-Cognitive Memory Bank baseline, and a plain π0 baseline, plus ablations on memory, HLP fine-tuning, and two-stage inference, with a failure-mode breakdown.

Main Findings

• Intra-episode gains: HeLM reaches 85.7% average subtask success versus Keyframe-HLP (72.9%), PCMB-π0 (39.5%), and Naïve-π0 (16.7%). Under strict task success (correct order and proper termination), HeLM retains 81.3%, while Keyframe-HLP falls to 29.3% and both other baselines fall to 0%.
• The headline improvement: HeLM improves intra-episode success by 12.8 percentage points over the strongest baseline.
• Inter-episode gains: HeLM averages 89.2% subtask success across E1–E5 (E1: 90.5, E2: 87.7, E3: 90.0, E4: 80.0, E5: 97.7), compared with 16.6% for Keyframe-HLP. PCMB-π0 and Naïve-π0 were excluded because they reset internal state at episode boundaries.
• Compositional generalization: On held-out variants A1′–A5′ with no additional HLP fine-tuning, HeLM averages 78.0% versus 65.7% for Keyframe-HLP, which additionally fails to detect termination on A1′, A4′, and A5′.
• Memory and fine-tuning ablations (LIBERO-Mem): HeLM averages 58.5% success versus 36.5% for π0.5, 12.0% for a memoryless HLP, and 8.5% for an unfine-tuned HLP. On the memory-intensive T3–T8 group, HeLM reaches 45.8% while π0.5 reaches 10.0% and both ablations reach 0%. This indicates the gains are not merely from added hierarchy.
• Two-stage inference matters: A one-stage variant that jointly predicts events, memory updates, and commands averages 67.7% versus 85.7% for HeLM, with the largest drop on A3 (22.2% versus 96.6%).
• Long-horizon robustness: HeLM maintains 100% success on A1 long as the repetition count grows to seven, and stays at or above 77% on A2 long as the required sequence grows to seven subtasks.
• Efficiency: Text memory scales better than stored images. At long horizons HeLM is 2.5× faster than the keyframe baseline and uses up to 4.5× less memory. Keyframe-HLP's peak VRAM reaches 28 GB for an 8-event episode, while HeLM stays under 6 GB.
• Failure modes: HeLM's failures are dominated by Detection (64.1%) and Low-Level execution (31.2%), with rare Order failures (4.7%) and no observed Termination failures. Most detection errors are false positives on drawer-opening and completion events, which corrupt memory in ways the framework cannot self-correct. Keyframe-HLP instead shows 33.3% Termination and 20.2% Detection failures.

Methodology in Plain English

The system is split into two levels. A high-level policy, built on a fine-tuned Qwen2.5-VL-7B model, reads the current image, the instruction, and a short text memory, then writes an action command such as which button to press. A low-level policy, a VLA model, turns that command plus the image into actual robot motion and executes a chunk of actions.

The high-level policy alternates between two modes that share the same weights and differ only in their system prompts. In "detect" mode it checks whether a task-relevant event has happened (for example, "pressed red button"). In "update" mode it rewrites the language memory and issues a new command. The update mode only fires when an event is detected or the instruction changes, so the memory is not churned on every frame. The high-level policy is invoked every 10 low-level steps.

To generate training data without recording long, hard-to-stage episodes, the authors define a TaskSpec: a per-task schema listing valid action commands, task-relevant events, and the memory update each event should cause. Short subtask recordings are then stitched logically and paired with synthesized memory states, so each training sample is just an image plus a memory-state prompt. The authors report these samples total 166,590 (about 130k train / 36k validation), and that the HLP was fine-tuned with QLoRA (rank r = 16) for 3,000 steps in about 15 hours on a single NVIDIA RTX A6000. Annotating event frames costs roughly 5 minutes per subtask, covering about 30 episodes.

Real-world tests used a 7-DoF AgileX PiPER arm with table- and wrist-view RGB cameras. Baselines all shared the same π0 action backbone, fine-tuned on 150–200 demonstrations per task, so low-level action capacity was held constant.

Why This Matters

• Research impact: The paper reframes memory representation, rather than model scale, as a first-class design choice for robot policies, and it argues that language is a natural memory format for VLM-based systems because it is compact, interpretable, and inspectable. It also introduces inter-episode persistence as a distinct evaluation axis that existing benchmarks such as LIBERO, RLBench, MemoryBench, LIBERO-Mem, RMBench, and RoboMME do not cover.
• Real-world applications:
  • Household cleaning, such as wiping a surface region by region without re-wiping finished areas.
  • Search and retrieval, such as opening drawers in sequence and stopping once an object is found.
  • Multi-step assembly or food preparation, where the robot must remember the current stage and prior placements.
  • Assistive and collaborative settings, where a robot must remember an object's original location or a human-demonstrated action order across separate interactions.
• Industry relevance: Because memory costs scale with language tokens rather than image tokens (under 6 GB versus 28 GB in the reported comparison), the approach is more practical for deployment on robots with limited onboard compute over long horizons. The event-triggered update scheme also reduces inference calls compared with continuously re-querying a large model.

Future Directions

• Open-ended event discovery: HeLM relies on predefined task-relevant events supplied through TaskSpec. Automatically deciding what is worth remembering, without hand-authored event vocabularies, is left open.
• Continuous state tracking: The evaluation covers discrete events. Tracking things like liquid levels or precise geometric history is not addressed.
• Higher-frequency or asynchronous high-level inference: The current synchronous chunk-level invocation adds overhead and may miss transient events that occur entirely inside one low-level action chunk.
• Memory organization ablations: The benefit of splitting working and episodic memory, versus a single persistent language memory, remains unevaluated by the authors' own account.
• Inter-episode baselines: The authors note that dedicated inter-episode baselines do not yet exist and that developing them is an open direction.

Target Audience

Robotics and embodied-AI researchers working on VLA models, long-horizon manipulation, and robot memory; engineers building manipulation systems that need reliable multi-step task execution; and readers interested in using language as an interpretable interface between high-level reasoning and low-level control. The paper is most useful to those with some background in imitation learning and vision-language models, though the central idea is accessible to newcomers.

Authors’ abstract

Recent advances in Vision-Language-Action (VLA) models have enabled fine manipulation capabilities, yet performing tasks that require working memory remains a key challenge. Most existing VLA architectures operate under a Markovian assumption, mapping only the current observation to an action, which leads to failure in non-Markovian environments. Inspired by the human 'inner monologue', we propose Hierarchical Explicit Language Memory (HeLM), a hierarchical VLA framework that maintains task-relevant history in explicit language through factorized event detection and event-triggered memory updates. We evaluate HeLM across 10 non-Markovian real-world tasks with 56 variants, including five newly formulated inter-episode tasks that require persistent context across task boundaries. HeLM improves intra-episode success by 12.8 percentage points over the strongest baseline and achieves 78% subtask success on compositional generalization tasks, while using up to 4.5x less memory and running 2.5x faster than the keyframe baseline at long horizons.