Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting

AI summary

Overview

• Research area: Robotics and robot learning — specifically Vision-Language-Action (VLA) models, instance-level personalization of robot policies, and visual prompting. Keywords listed by the authors: Machine Learning, ICML.
• Technical level: Advanced. The paper assumes familiarity with VLA policies (π₀, π₀.₅), open-vocabulary detection, embedding-based matching, segmentation and tracking, and token-learning personalization baselines.
• Scope (one sentence): The paper proposes Visual Attentive Prompting (VAP), a training-free input-side adapter that lets a frozen VLA manipulate a user's specific personal object — such as "my cup" — given roughly five reference images, and evaluates it on two new simulation benchmarks and a real-world SO-101 tabletop setup.

What This Paper Is About

General-purpose VLA models follow generic commands well ("pick up the cup") but collapse personalized requests such as "bring my cup" to category-level recognition, because user-specific instances and possessive instructions are rarely seen during training. The authors formalize personal object manipulation: given a small reference set of photographs of the actual object (K ≈ 5) plus a possessive instruction naming its coarse category, a frozen policy must pick out that exact instance among visually similar same-category distractors and act on it. Their goal is to achieve this with no per-object fine-tuning or parameter updates, by intervening only on the policy's inputs.

Key Contributions

1. A personalization task for VLAs. The authors introduce the setting where a policy must manipulate user-specific objects unseen during training, among visually similar distractors, using only a few reference images, with the actual instance treated as the primary disambiguating signal rather than language.
2. Visual Attentive Prompting (VAP). A training-free input adapter that couples reference-based instance grounding with a mask-aligned highlight and a matched instruction rewrite, so a frozen VLA acts on the correct instance. The paper frames this as a zero-shot approximation to searching for the visual intervention that maximizes policy success.
3. Two simulation benchmarks and a real-world setup. Personalized-SIMPLER (Google Robot / Fractal and WidowX / Bridge) and Personalized-VLABench (Franka), plus a real SO-101 tabletop benchmark, covering multiple robots and task types.
4. Ablations and analysis of why input-side grounding is needed. The paper shows that a strongly optimized token-learning baseline still improves only marginally, and that alternative visual prompt styles (points, trajectory arrows, numbered markers) fall far short of mask-aligned tinting.

Main Findings

• VAP beats generic and language-only baselines on Personalized-SIMPLER (Google Robot). On the pen-holder task in the visual-matching track, VAP raises SR/CMR from 8.5%/10.5% (generic π₀) to 60.3%/89.2%. These gains persist under variant aggregation (SR 58.2%, CMR 87.3%). Hard Prompts remain competitive on the simpler bottle task, but VAP is the most consistent across scenarios.
• The WidowX (Bridge) results show the same pattern. On Task 3 (Pick shaver) VAP reaches 71.3% SR where baselines struggle to locate the target; on Tasks 4 and 5 it attains >90% SR (92.1% and 95.0% SR respectively, with 92.1% and 100.0% CMR). On Task 6 VAP reaches 100.0% CMR and 75.6% SR, where baselines show high CMR but low SR (e.g., generic π₀ 80.6% CMR, 27.8% SR).
• VAP leads all multi-view selection scenarios on Personalized-VLABench. It surpasses the strongest baseline by +36.8 points on the leather bag task (89.2 vs. 52.4). Its advantage is smaller but present on the shoe, cat figurine, miniature house, and cup tasks (54.0, 51.6, 52.4, 60.8 respectively).
• Real-world results confirm the simulation trend. On four selection tasks, text-based baselines plateau at 40.0–45.0% SR while VAP reaches 80.0% SR (versus 30.0% for generic π₀.₅). On four pick-and-place tasks, VAP improves average SR from 18.8% to 58.8%, while soft/hard prompts stay in the 27.5–31.2% range. VAP also closes the CMR–SR gap that baselines exhibit.
• Language-centric and token-learning baselines fall short. Hard Prompts (Short/Long) and a Yo'LLaVA-style Soft Prompt baseline provide only partial or marginal gains, even though the Soft Prompt baseline was deliberately strengthened with oracle hard negatives and reaches >95% accuracy on VQA recognition probes.
• Failure modes differ by setting. In single-view Personalized-SIMPLER, failures are dominated by control limitations (Case 3: 86.9% for Fractal, 78.6% for Bridge of failures), not visual ambiguity (Case 1 under 22%). In multi-view benchmarks (Personalized-VLABench, real world), cross-view inconsistency dominates (Case 2: 59.2% and 50.9% of failures). Overall failure rates are 37.6% (Fractal), 16.6% (Bridge), 38.4% (Personalized-VLABench), and 31.9% (real world).
• Overhead is minimal. Initialization costs 0.26 s total (Grounding DINO 0.19 s; segmentation and embedding 0.07 s). Per control step, SAM2 tracking adds 0.02 s against 0.20 s for VLA policy inference — about 10% overhead to the control loop.
• Mask-aligned tinting is the key prompt style, not just any visual prompt. On the harder WidowX setting, mask-aligned identity tinting reaches 83.5% SR, while points, trajectory arrows, and numbered markers stay between 28 and 36%.

Methodology in Plain English

VAP never touches the policy's weights. It edits what the policy sees and reads at each timestep.

First, grounding. The user registers a few reference photos per personal object. The system detects the target in each reference image with a category detector, crops around the box to reduce background bias, and embeds each crop with a frozen visual encoder (DINOv2, using the final-layer ℓ₂-normalized [CLS] token). At test time, the personalized instruction ("bring my cup") is parsed to extract the generic category ("cup"), which is used as the text query for an open-vocabulary detector (Grounding DINO) on every camera view. Each reference embedding then "votes" for the most similar detected proposal; the proposal with the most votes wins, with ties broken by mean cosine similarity. That box is refined into a pixel-wise mask with a class-agnostic segmenter, and for later timesteps a real-time tracker (SAM2) propagates the mask using its memory state instead of re-running detection. If no candidates are detected (N_v = 0) or the tracker mask is unavailable, VAP passes the unmodified view through.

Second, visual prompting. The mask is turned into a semi-transparent, mask-aligned tint over the target, leaving the background and the proprioceptive state unchanged. In parallel, the instruction is rewritten by template string matching, replacing "my X" with "the tint-color X" (for example, "pick up my cup" becomes "pick up the red object"). The mask and the rewrite are designed as one binding mechanism: the mask says where, the rewrite supplies the language token that tells the policy which anchor to attend to. The paper reports that without either signal, the policy reverts to its category-level prior.

Evaluation uses a shared frozen backbone for VAP and all baselines: π₀.₅ unless otherwise stated, with π₀ checkpoints for the Fractal and Bridge settings of Personalized-SIMPLER (following prior work showing π₀ is strong there). π₀.₅ is fine-tuned only for environment adaptation on generic data that explicitly excludes personal objects and personalized instructions, then frozen. Metrics are Success Rate (SR) and, for pick and pick-and-place tasks, Correct Movement Ratio (CMR) — the fraction of episodes in which the policy moves the target at least once regardless of final success. CMR is not defined for pointing or selection tasks. Perception modules are DINOv2, Grounding DINO, and SAM2.

Why This Matters

• Impact on research: The paper argues that personalization in embodied control is an instance-level problem, not a language problem. It shows that a state-of-the-art token-learning approach adapted from vision-language personalization (Yo'LLaVA-style) does not transfer to closed-loop action: even when the learned token recognizes the object in static images and induces object-centric attention, the induced attention drifts across frames. External, mask-based re-anchoring at every frame sidesteps this, and VAP's modular factorization makes remaining grounding errors explicitly diagnosable rather than silent.
• Real-world applications (drawn from the paper's framing):
  • Fetching a user's own coffee mug or water bottle when lookalikes are in the same cabinet or table.
  • Retrieving a pet's favorite toy from the living room floor.
  • Selecting a specific item among same-category belongings, such as "my slipper" or "my leather bag."
  • Longer-horizon household chores where a specific personal object must be placed into a receptacle, such as "put my scrubber into the bowl."
• Industry relevance: The method requires no per-object training, so a new personal object becomes usable immediately after it is registered with a few reference photographs — relevant to deployment economics, where per-instance fine-tuning is impractical. The 10% control-loop overhead and the design that inherits future improvements in open-vocabulary detection and segmentation without policy-side retraining are the practical selling points. Robotics platforms referenced include the Google Robot, WidowX, Franka, and the LeRobot SO-101 arm.

Future Directions

• Explicit cross-view association or evidence fusion. VAP factorizes grounding and tracking per camera view for real-time inference, so occlusions or partial visibility can make per-view matching lock onto different same-category instances. The paper proposes enforcing agreement on a single target identity before prompting, and leaves the implementation to future work.
• Reducing the dependence on segmentation and multi-view mask consistency, which the authors identify as the framework's current contingency.
• Extending the paradigm to more complex personalization, specifically user-specific spatial preferences and multi-step long-horizon tasks.
• Closing the control-side bottleneck. In single-view settings, most failures come from the VLA's physical manipulation capability rather than from grounding, so improved control (rather than better perception) is what limits further gains there.

Target Audience

Robotics and embodied-AI researchers working on VLA policies, robot manipulation, and robot personalization; practitioners who need to deploy general-purpose policies on user-specific objects without retraining; and researchers in multimodal grounding, open-vocabulary detection, and visual prompting who are interested in how perception-side interventions transfer to closed-loop control. Readers evaluating benchmark design for instance-level (rather than category-level) generalization will also find the two new simulation benchmarks and the real-world protocol directly useful.

Authors’ abstract

While Vision-Language-Action (VLA) models generalize well to generic instructions, they struggle with personalized commands such as "bring my cup," where the robot must act on one specific instance among visually similar objects. We study this setting of manipulating personal objects, in which a VLA must identify and control a user-specific object unseen during training using only a few reference images. To address this challenge, we propose Visual Attentive Prompting (VAP), a simple-yet-effective training-free perceptual adapter that equips frozen VLAs with top-down selective attention. VAP treats the reference images as a non-parametric visual memory, grounds the personal object in the scene through open-vocabulary detection and embedding-based matching, and then injects this grounding as a visual prompt by highlighting the object and rewriting the instruction. We construct two simulation benchmarks, Personalized-SIMPLER and Personalized-VLABench, and a real-world tabletop benchmark to evaluate personalized manipulation across multiple robots and tasks. Experiments show that VAP consistently outperforms generic policies and token-learning baselines in both success rate and correct-object manipulation, helping to bridge the gap between semantic understanding and instance-level control.