Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective

AI summary

Overview

Research area: Robotics / embodied AI — specifically vision-language-action (VLA) models, object-centric representation learning, and partially observable robotic manipulation benchmarks.

Technical level: Advanced. The paper combines a new benchmark suite with a slot-based state-space model (SSM) formulation, attention-based object binding, and VLA action decoding.

Scope: The paper introduces LIBERO-Mem, a ten-task non-Markovian manipulation benchmark targeting object-level partial observability, and proposes Embodied-SlotSSM, a slot-centric VLA framework designed to keep persistent, spatio-temporally consistent object memory for temporally scalable action prediction.

What This Paper Is About

Most robotic visuomotor policies decide the next action from the current camera frame alone, which fails whenever the same image can correspond to different underlying task states. The authors argue that solving everyday tasks — where objects look identical, get occluded, or must be manipulated a specific number of times — requires memory of each object's interaction history rather than reactive perception. Their goal is to expose this gap with a dedicated stress-test benchmark (LIBERO-Mem) and to offer an initial memory-centric architecture (Embodied-SlotSSM) that reasons over object-specific history.

Key Contributions

1. LIBERO-Mem benchmark. A new non-Markovian robotic manipulation benchmark with ten tasks that systematically evaluate memory-augmented models on long-horizon manipulation, emphasizing object permanence, historical reasoning, and structured memory retention. It covers four object-centric memory dimensions: Object Motion (OM), Object Sequence (OS), Object Relations (OR), and Object Occlusion (OO).

2. Embodied-SlotSSM framework. A slot-based state-space modeling approach that encodes persistent, object-centric memory representations, enabling structured tracking and decision-making under partial observability. It combines slot-state-space modeling for short-term history reconstruction with a relational encoder that aligns tokens to action decoding.

3. Benchmark design features absent in prior suites. Per the paper's comparison table, LIBERO-Mem uniquely includes subgoal-aware evaluation, object identity ambiguities, object and subgoal annotations, and stress-testing by temporal scaling — none of which are listed for MemoryBench, MIKASA-Robo, LIBERO, or RLBench. MIKASA-Robo is described as spanning 32 tasks but lacking long-horizon trajectories.

4. Oracle-supported empirical instantiation. The authors evaluate a version of their model, Naive E-SlotSSM, that uses oracle text-embedding subgoals rather than inferring them autonomously, across both general Markovian tasks (LIBERO-Goal) and the non-Markovian LIBERO-Mem suite.

Main Findings

• LIBERO-Mem task composition: The suite contains 10 tasks. Repetition tasks require lifting and placing a bowl or bottle 3, 5, or 7 times; swap tasks involve 2 or 3 bowls and an empty plate; occlusion tasks involve placing a bowl in the closest basket and then moving a basket to the middle.

• Data scale: Each task contains 200–700 frames and 120 collected trajectories, of which 100 are refined as training data and 20 are held out for validation. Expert demonstrations were collected via smooth keyboard control with multi-key tracking.

• General task performance (LIBERO-Goal, Table 3): Naive E-SlotSSM achieves the highest average success rate at 83.0%, versus 75.5% for SlotVLA with horizon h=8 and 32% for SlotVLA with h=1. Per-task, Naive E-SlotSSM reaches 100% on "bowl in stove" and "stove turned on," 95% on "plate on front-of-stove," 90% on "bowl in cabinet," "bottle on cabinet," and "bowl on plate," 75% on "cheese in bowl" and "bottle on rack," 70% on "top drawer open → bowl in," and 45% on "mid drawer open."

• Token efficiency framing: OpenVLA encodes entire video sequences using 256 dense tokens, while the authors' customized object-centric VLA design operates with only 16 slot tokens; however, tokens scale linearly with slot and sequence dimension, which the authors cite as driving intractable memory and compute costs in long-horizon settings.

• Non-Markovian subgoal completion (LIBERO-Mem, Table 4): Embodied-SlotSSM (Naive E-SlotSSM, token count 32) achieves an average subgoal completion of 14.8%, compared with 5.0% for π₀ (h=1, 256 tokens), 0% for SlotVLA (h=1, 16 tokens), and 5.0% for SlotVLA (h=8, 128 tokens).

• Where the gains appear: Naive E-SlotSSM reaches 50.0% on T1 (pick and place bowl once), 33.3% on the repeated (3x) version, 14.3% on T5, 30% on T9, and 20% on T10. It records 0% on T2, T4, T6, T7, and T8. π₀ reaches 50.0% on T1 and 0% on all other reported tasks; SlotVLA at h=8 reaches 50.0% on T1 and 0% elsewhere; SlotVLA at h=1 records 0% on every reported task.

• Qualitative tracking: Slot attention visualizations on task T1 show the model maintaining consistent attention to target objects over time, which the authors interpret as evidence of object permanence and tracking relevant to long-horizon reasoning.

• Acknowledged shortfall: The authors state that persistent memory modeling remains modest because the empirical version relies on an object-level subgoal monitor, and that subgoal grounding remains a key open challenge.

Methodology in Plain English

The authors first define the failure mode formally. Standard VLA training predicts the next action from the current observation and language instruction, which assumes the process is Markovian. The paper describes non-Markovian settings as cases where two timesteps have near-identical visual inputs but require different correct actions because their histories differ. LIBERO-Mem is built so that at least two trajectories for a task contain identical observations paired with different underlying task states, forcing any policy to consult history.

On the modeling side, the architecture has three linked pieces. First, Slot Attention converts dense visual features into a fixed set of modular object tokens (K=16 by default) by iteratively binding spatial patches to learnable queries. To keep the same slot attached to the same object across frames, slots at each timestep are initialized randomly only at t=0 and otherwise from the final refined slots of the previous timestep. A temporal contrastive loss, using cosine similarity with temperature τ=1, pulls a slot toward its counterpart in a nearby frame and pushes it away from slots from other videos or locations.

Second, a SlotSSM module applies an input-dependent state-space recurrence with block-diagonal, slot-conditioned transition matrices, so each object keeps its own hidden state. Rather than predicting a single next embedding, it predicts a window of P = p + q static latents spanning p past and q future object representations around the current timestep, which provides supervision for both forward dynamics and backward reconstruction.

Third, action decoding uses a Slot Fusion module that combines each current slot, its predicted next slot, and (in the empirical version) an oracle subgoal embedding. A lightweight Relation Encoder cross-attends the fused slot latents with raw visual features to produce relational tokens (16 relational tokens for K=16 slots). The final action is decoded conditioned on the relation tokens, the slot dynamics, and the task query embedding. The evaluated variant is named Naive E-SlotSSM because it uses oracle subgoal representations instead of discovering them itself.

Evaluation compares against π₀ and against an object-centric version of SlotVLA built on OpenVLA with slot attention, each at specified input-frame horizons, over N=20 seeds. Success rate is the ratio of successful completions to total attempts; on LIBERO-Mem, the reported metric is subgoal completion, the number of completed subgoals divided by total subgoals across seeds.

Why This Matters

Impact on research. The paper argues that existing robotic benchmarks — RLBench, LIBERO, and RoboCasa — are built under the Markovian assumption, while memory-focused efforts like MemoryBench and MIKASA-Robo operate in simplified settings without object-level ambiguities or temporal scaling. LIBERO-Mem is positioned as a targeted instrument for exposing the absence of structured memory and temporal reasoning in current VLA architectures, and Embodied-SlotSSM as an initial object-centric solution.

Real-world applications:

• Cooking and kitchen assistance, where an agent must remember whether it already added an ingredient or where it last placed a container.
• Laboratory automation involving repeated, ordered steps with visually similar vessels.
• Industrial assembly with sequenced subgoals and part occlusion.
• Household or warehouse rearrangement, where identical-looking objects must be distinguished by their interaction history.

Industry relevance. The authors frame the contribution as advancing memory-aware, object-centric visuomotor systems that help robots track task structure and avoid redundant or missed actions. The token-scaling discussion — 16 slot tokens versus 256 dense tokens, with linear growth in slot and sequence dimensions — makes temporal scalability a practical engineering concern for deploying memory-based VLAs on long episodes.

Future Directions

• Self-discovered subgoals. The authors explicitly flag that Naive E-SlotSSM uses oracle subgoal representations rather than inferring them autonomously, leaving open the challenge of self-discovered subgoal reasoning.
• Object-level POMDP capability. The limitations section states that POMDP reasoning at the object level remains an open problem beyond the current empirical implementation.
• Physical extension. LIBERO-Mem is described as a simulated setting intended for future physical extension, and the authors characterize Naive E-SlotSSM as a weak baseline.
• Temporal scalability of memory-based VLAs. Since tokens scale linearly with slot and sequence dimension, developing scalable strategies for long-horizon memory remains a stated motivation.

Target Audience

Robotics and embodied-AI researchers working on VLA models, memory-augmented policies, and partially observable manipulation; benchmark designers interested in non-Markovian task suites and subgoal-level evaluation; and object-centric representation learning researchers extending slot-based methods from perception to long-horizon control. Readers should be comfortable with state-space models, attention mechanisms, and POMDP terminology.

Authors’ abstract

As embodied agents operate in increasingly complex environments, the ability to perceive, track, and reason about individual object instances over time becomes essential, especially in tasks requiring sequenced interactions with visually similar objects. In these non-Markovian settings, key decision cues are often hidden in object-specific histories rather than the current scene. Without persistent memory of prior interactions (what has been interacted with, where it has been, or how it has changed) visuomotor policies may fail, repeat past actions, or overlook completed ones. To surface this challenge, we introduce LIBERO-Mem, a non-Markovian task suite for stress-testing robotic manipulation under object-level partial observability. It combines short- and long-horizon object tracking with temporally sequenced subgoals, requiring reasoning beyond the current frame. However, vision-language-action (VLA) models often struggle in such settings, with token scaling quickly becoming intractable even for tasks spanning just a few hundred frames. We propose Embodied-SlotSSM, a slot-centric VLA framework built for temporal scalability. It maintains spatio-temporally consistent slot identities and leverages them through two mechanisms: (1) slot-state-space modeling for reconstructing short-term history, and (2) a relational encoder to align the input tokens with action decoding. Together, these components enable temporally grounded, context-aware action prediction. Experiments show Embodied-SlotSSM's baseline performance on LIBERO-Mem and general tasks, offering a scalable solution for non-Markovian reasoning in object-centric robotic policies.