CondenseVLA: Learnable History Condensation for Efficient Multi-Frame VLA

AI summary

Overview

Research area: Computer Vision / Robot Learning — specifically vision-language-action (VLA) policies for robot manipulation, with a focus on efficient temporal (multi-frame) context modeling.

Technical level: Advanced. The paper assumes familiarity with transformer attention, diffusion/DiT action decoders, slot-attention style bottlenecks, and standard manipulation benchmarks (SimplerEnv, LIBERO, RMBench).

Scope: The paper introduces CondenseVLA, a decoder-side module that compresses cached multi-frame VLM features into a small fixed set of learned "slot" tokens for action decoding, and evaluates it on simulated, memory-dependent, and real-robot manipulation tasks.

What This Paper Is About

Robot manipulation often hinges on short-lived visual cues — a brief contact, a temporary occlusion, an object that was moved a moment ago — that a single camera frame cannot disambiguate. Simply stacking more past frames into a VLA policy is expensive (the paper cites HAMLET's report that appending only four extra past frames can slow the forward pass by about 35%) and often unhelpful because neighboring frames are highly repetitive and fixed-interval sampling can miss brief but important events. CondenseVLA's goal is to keep a bounded, learnable summary of recent visual history that conditions the action decoder, rather than enlarging the history window or hand-picking which frames to keep.

Key Contributions

1. A learnable history condensation module (LHC). A decoder-side module inserted between a frozen vision-language backbone and a diffusion action decoder that distills a length-M FIFO cache of per-frame VLM features into K slot tokens (K ≪ M) using learnable slot queries and multi-head cross-attention.
2. Auxiliary regularization for slot diversity. Two simple penalties: an attention-diversity loss that penalizes overlap among slot attention maps, and a slot-decorrelation loss (cosine similarity, applied only to "middle-only" slot pairs) that discourages redundant slot embeddings.
3. Empirical results across four settings. Simpler-WidowX success improves from 58.7% to 69.8%, LIBERO reaches 97.2%, real-world Galaxea R1 Lite reaches 52.8%, and RMBench reaches 21.6%.
4. Analysis of why condensation beats naive history expansion. Component ablations, a K/M scaling study, latency measurements, and comparisons against static/strided history-selection templates, showing that learned condensation outperforms rigid sampling at matched history and output budgets.

Main Findings

• Simpler-WidowX: CondenseVLA reaches 69.8% mean success versus 58.7% for the CronusVLA baseline (+11.1 percentage points). Task-level gains are +13.9 / +15.3 / +11.1 / +4.2 pp for Spoon on Towel, Carrot on Plate, Stack Cube, and Eggplant in Basket respectively; the paper explicitly notes the benefits are non-uniform and that these point estimates do not establish task-wise statistical significance.

• LIBERO: 97.2% mean success versus 96.3% for CronusVLA on the same external-and-wrist RGB/no-state protocol; on the Long suite, 94.6% versus 94.0%. The authors describe these scores as near-saturated and state they do not by themselves isolate memory-dependent reasoning.

• RMBench (memory-dependent): CondenseVLA reaches 21.6% overall versus 14.3% for dense CronusVLA (+7.2 pp), and 4.7% for a current-frame-only OpenVLA reference. Gains are larger on the single-observation M(1) category (28.8% vs 18.2%, +10.6 pp) than on the multi-observation M(n) category (12.5% vs 9.5%, +3.0 pp). All three methods fail on Press Button.

• Real-world (Galaxea R1 Lite): 52.8% mean final success versus 45.2% for CronusVLA (+7.6 pp). Multi-stage tasks improve from 37.3% to 46.7% (+9.3 pp). Per task: Pick and Place Twice 40% → 50%, Swap Cubes 50% → 62%, Put Objects in Drawer 22% → 28%.

• Learned condensation beats fixed selection: Under the ablation protocol at M=20, K=6, learned condensation scores 70.8%, versus 59.4% for strided sampling (six positions [0,4,8,11,15,19]), 55.2% for static-index selection, and 54.2% for static-weight selection — gaps of −11.4, −15.6, and −16.6 pp.

• History length and slot budget interact: At M=20, K=6 is best (70.8%); at M=30, K=10 is best (64.6%). No separate held-out hyperparameter split was used, and the M=20/K=6 Simpler setting was transferred unchanged to RMBench.

• Latency is not uniformly better: Measured policy latency is 0.25% higher than dense conditioning at M=10, but 3.8% lower at M=20 and 7.4% lower at M=30. The authors state the results do not establish a universal end-to-end speedup, since LHC cost still grows with M.

• A stated limit on memory: Because history is a FIFO window with no explicit retrieval path, persistent semantic task state, or high-level planner, performance on multi-observation RMBench tasks stays low — once an event leaves the window, the mechanism cannot recover it.

Methodology in Plain English

The policy is built on CronusVLA, a multi-frame OpenVLA variant with a DiT-style diffusion action decoder. The vision encoders and language model stay frozen; fine-tuning only updates the multimodal projector, the new condensation module, and the action decoder.

At each environment step, the VLM encodes only the current observation once. The M most recent frame features sit in a first-in-first-out buffer. A condensation module takes K learnable "query" vectors and has them attend over those cached M features, producing K slot tokens that summarize recent history. Each slot query is built from a learned base query, a learned slot embedding, and a fixed relative scalar between 0 and 1 (a weak identity signal that breaks symmetry, not a hard temporal assignment). A learned history-position embedding enters only the value path, with its scale initialized to zero, so keys carry no explicit position bias.

The diffusion decoder is then conditioned on the K slot tokens concatenated with the current frame feature — a fixed-length input independent of M. Historical features are stop-gradient during training, so gradients flow only through the current-frame path.

Training combines three losses: the standard diffusion noise-prediction loss, an attention-diversity loss that pushes the row-normalized, head- and batch-averaged attention matrix toward the identity (penalizing overlap between slots), and a slot-decorrelation loss that penalizes squared cosine similarity between "middle-only" slot pairs. Weights are λ_div = 0.06 and λ_slot = 0.001, with learning rate 2 × 10⁻⁵, AdamW, no weight decay, no warm-up, gradient clipping at 1.0, and seed 42.

Configuration differs by benchmark: M=20, K=6 on Simpler-WidowX, RMBench, and the real robot; M=10, K=3 on LIBERO. LHC uses four attention heads with D=4096 and d_h=1024, and has 100.8M parameters — about 1.35% of the 7.47B frozen backbone and 74.8% of the 134.7M decoder. Decoder cost is O(KMD) for the attention mixing, and the K+1 input features become K+6 decoder-conditioning tokens.

Why This Matters

Research impact. The paper reframes the multi-frame VLA question from "how long should the history be?" to "how should history be compressed?" It provides a controlled dense-versus-condensed comparison (matched initialization, data, steps, seed, action horizon, and evaluation protocol) and shows that a small, learned bottleneck can beat dense conditioning at the same history length — while also candidly documenting where a bounded FIFO window fails.

Real-world applications:

• Long-horizon pick-and-place and rearrangement in warehouses, where a robot must remember which bin or plate it already used in a multi-step sequence.
• Drawer and cabinet manipulation (open, insert, close) where success depends on intermediate states that are occluded from the current viewpoint.
• Memory-dependent assistive or service robotics, where a robot must retain a single past observation — such as where an object was last seen — to act correctly.
• Deployment on latency- and compute-constrained robot hardware, where the fixed decoder-conditioning budget bounds inference cost as the history window grows.

Industry relevance. Because the method leaves the VLM backbone frozen and adds a small module at the decoder interface, it is a retrofit onto existing multi-frame VLA stacks rather than a retraining-from-scratch approach. The reported parameter overhead (100.8M, ~1.35% of the backbone) and the K/M trade-off table give practitioners a concrete knob for balancing temporal coverage against latency.

Future Directions

• Beyond the FIFO window. The authors identify the bounded recent window as the main limitation and point to the need for explicit retrieval of events that have left the buffer, persistent semantic task state, or a high-level planner to lift performance on multi-observation RMBench tasks.
• Principled selection of M and K. The current choices came from Simpler and were transferred unchanged, with no separate held-out tuning split. A method for selecting the history length and slot budget per target task — and evidence that these transfer robustly — remains open.
• Separating selectivity from added capacity. The paper notes that LHC adds trainable capacity, so the gains cannot be attributed solely to temporal selection; controlled experiments that match capacity would sharpen the claim.
• Broadening the evidence base. The evaluation uses one VLA backbone family and one physical robot platform, and the paper also notes that condensation cannot recover evidence absent from the frozen visual features.

Target Audience

Researchers and engineers working on robot learning and embodied AI — particularly those building or deploying VLA policies that need temporal context under compute constraints. It is most useful to readers already comfortable with transformer attention, diffusion policies, and manipulation benchmarks. Readers new to the field will find the introduction and the plain-language motivation (the Pick and Place Twice and Swap Cubes failure cases) accessible, but the method and ablation sections assume substantial background. Practitioners considering a multi-frame upgrade to an existing OpenVLA/CronusVLA-style stack are the most direct beneficiaries.

Authors’ abstract

Robot manipulation often depends on short-term cues - such as brief contacts, temporary occlusions, and intermediate object states - that are ambiguous in a single frame and can only be resolved by integrating observations over time. However, injecting historical information solely by enlarging the history window is computationally costly and frequently yields limited gains due to substantial redundancy and task-agnostic, inflexible sampling of observations. We propose CondenseVLA, a multi-frame VLA policy with a learnable decoder-side history condensation module for using short-term visual context in action decoding. Given a sequence of cached per-frame features from the VLM backbone, CondenseVLA distills observation history into a fixed set of slot tokens using learnable queries and multi-head cross-attention. Simple auxiliary regularization penalizes overlap among slot attention maps and decorrelates slot embeddings, encouraging complementary attention and reducing slot redundancy. Across the evaluated tasks, CondenseVLA improves Simpler-WidowX success from 58.7% to 69.8% (+11.1 percentage points), achieves 97.2% on LIBERO, and reaches 52.8% on real-world tasks. It achieves 21.6% success on memory-dependent RMBench tasks. These results support selective condensation of bounded recent visual evidence for action decoding, while harder multi-observation tasks expose the limits of a finite history window. Project page: https://github.com/FeiyangHong/CondenseVLA