PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training

AI summary

Overview

Research area: Robot learning — specifically vision-language-action (VLA) policies, imitation learning, and shortcut learning in multimodal models.

Technical level: Advanced. The paper assumes familiarity with VLA architectures, behavior cloning, flow matching, mutual information, and robot evaluation protocols.

Scope: The paper identifies a class of failure modes called "modality shortcuts" in VLA policies, proposes a three-part training-data intervention called PerturbBot to reduce them, and introduces an offline diagnostic score called GroundFscore that measures the severity of shortcut reliance without robot rollouts.

What This Paper Is About

VLA policies can succeed at manipulation tasks while ignoring the evidence that should actually determine their behavior — an object near the wrist camera can hijack a reach, a familiar noun can trigger a trained action even after the verb changes, and a gripper that closed on nothing may lift anyway. Because successful demonstrations contain these regularities, behavior cloning rewards agreement with the expert action rather than the evidence used to predict it, and adding more demonstrations of the same kind can raise task success without removing the shortcuts. The paper's goal is to change what is scaled in training data so that task-relevant evidence becomes easier to use and shortcuts become insufficient on their own, while leaving inference unchanged.

Key Contributions

1. Identification of modality shortcuts in VLA learning. The authors unify three failure modes — salience capture (what to manipulate), noun lock-in (which semantics apply), and motor inertia (how to move) — and formalize why scaling demonstrations that preserve the same correlations need not remove them, using a mutual-information decomposition into redundancy and degeneracy.

2. PerturbBot: three independently ablatable data interventions. Task-preserving wrist-view perturbations (V), decision-relevant caption enrichment (C), and caption-grounded trajectory expansion with relabeled random-motion and failed-execution segments (R). Each keeps action supervision valid, changes what is scaled, adds no extra model or forward pass at deployment, and can be individually disabled.

3. GroundFscore: an offline shortcut-diagnosis score. A paired-edit metric combining a spurious response term (movement when nothing should move) and a causal sensitivity term (movement when something should move), computed as a harmonic mean so that a policy which simply ignores an input scores zero. It requires only forward passes, not rollouts, so it can be computed for every checkpoint.

4. A controlled training-and-evaluation framework. All eight combinations of V, C, and R are evaluated on 50 real-robot General PnP cases, extended to the 50-task RoboTwin 2.0 suite, with scaling ablations and a validity test of whether GroundFscore predicts online failures.

Main Findings

• The full method doubles baseline success. With a π 0.5 backbone on 50 real-robot General PnP cases, the baseline fine-tune reaches 42% success and a 2.71 mean progress score; PerturbBot (V + C + R) reaches 84% success and 4.36 progress.

• Each component raises mainly its own target readout. V lifts GF_V from 0.28 to 0.51 and cuts salience-capture failures from 68% to 36%; C lifts GF_L from 0.24 to 0.47; R lifts GF_A from 0.19 to 0.44 and cuts motor-inertia failures from 82% to 44%. Under the full method the readouts are GF_V 0.62, GF_L 0.57, GF_A 0.53, with SC at 20% and MI at 24%.

• Gains compound, and removal is costly. Every pair beats both of its members, and removing any single component costs 16–20 points of success.

• Generic controls stay near baseline. Noise/dropout at the V budget scores 44% success, twice the demonstrations scores 48%, negative guidance scores 46%, and wrist-camera removal scores 26%. Camera removal attains the lowest salience-capture rate (12%) only by losing 16 points of success and dropping GF_V to 0.16.

• PerturbBot scales faster and more healthily. On nested subsets of 125, 250, 500 (1×), and all 1,000 General PnP demonstrations at fixed optimizer steps over three seeds, PerturbBot gains 16.2 points of success per doubling versus 6.0 for fine-tuning, and 0.100 mean GroundFscore per doubling versus 0.013. At 1/4× it already matches fine-tuning at 2×, with eight times fewer demonstrations. From 1/4× to 2×, fine-tuning adds 18 points of success while mean GroundFscore moves only 0.04; PerturbBot raises success by 48 points and mean GroundFscore by 0.30, with both failure rates falling at every budget.

• Success and GroundFscore correlate far more tightly under PerturbBot. Spearman ρ = 0.93 for PerturbBot versus 0.41 for fine-tuning, whose mean GroundFscore barely varies so seed noise dominates its ranking.

• PerturbBot generalizes beyond pick-and-place. On RoboTwin 2.0's 50 bimanual tasks, the π 0.5 baseline scores 82.7 clean / 76.8 randomized (Full) and 70.7 clean / 46.0 randomized (Clean2Random). PerturbBot reaches 85.6 / 81.9 and 73.4 / 58.2. The Clean2Random gain on randomized scenes is 12.2 points versus 2.7 on clean scenes — the best randomized-scene result among the listed systems. Removing V costs 8.4 and 2.6 points on randomized scenes in the two settings.

• Controllability is preserved, not traded away. PerturbBot picks the distractor in 20% of target-selection probes versus 68% for fine-tuning, yet still selects the instructed object in 90%. Without a wrist camera a policy avoids the distractor more often (12%) but finds the instructed object in only 48%. PerturbBot lifts secured objects nearly as often as fine-tuning (92% vs. 94%) but lifts after empty grasps in 24% rather than 82%.

• GroundFscore predicts online failures better than success does. Across 36 checkpoints, its Spearman ρ with negative online failure rate is 0.72, versus 0.31 for task success; the advantage of 0.41 has a 95% cluster-bootstrap interval of [0.12, 0.68]. Stability alone (1 − R) reaches 0.44 but ranks the no-wrist policy above the baseline, and responsiveness alone (S) reaches only 0.38.

Methodology in Plain English

The authors start from a diagnosis rather than an architecture change. They argue that shortcuts arise for two reasons visible in the training data: collinearity, where the shortcut and the evidence always agree (the target fills the wrist view because the operator reaches for it; a noun always appears with one verb), and degeneracy, where the decision never varies (nothing follows gripper closure but a lift). More of the same demonstrations preserves both, which is why scaling alone cannot separate the shortcut rule from the intended rule.

Their fix is to change what the training data demand, in three ways, all applied at training time only:

• Visual (V): insert a plausible distractor off the grasp path in the wrist view, without hiding the target or changing contact geometry, so the recorded action stays valid. Edits that would change the target, collision constraints, or required grasp are rejected. The lesson is that a salient object need not be the instructed one, while wrist evidence should still guide the motion.
• Caption (C): use a vision-language model to split recordings into atomic skills (grasp, lift, move, put down) and describe each by acting hand, open/close, operation, object with just enough attributes to single it out, and its position, state, and grasp point. Brief and detailed instructions are mixed under the same rules so caption style does not reveal the data source, and the deployed policy still receives ordinary instructions.
• Trajectory (R): take recorded random motion and failed executions, split them into segments with synchronized images, motor context, and actions, discard unusable ones, and relabel each retained segment with a brief instruction for what it actually does plus a detailed version from C. A failed bowl pickup may contain the valid segment "move the empty gripper left and open it." The authors are explicit that relabeling adds behavioral coverage but cannot create an unrecorded recovery.

For evaluation, they avoid robot rollouts by pairing each input with two kinds of edits per shortcut family: a null edit that should leave the action unchanged (off-path decoy, paraphrase, another approach to the same pose) and a causal edit that should change the action by a known target amount computed from two expert recordings. The score is the harmonic mean of not-moving-when-you-shouldn't (precision) and moving-when-you-should (recall), so ignoring an input yields zero.

Experimentally, all conditions share initialization, optimizer, training steps, and batch size, so each comparison varies only the training data. The 50 evaluation layouts on the real robot are generated once by rule-based random placement and restored before each rollout, so every policy sees the same cases once. Captions are generated from the wrist-view perturbation prompt described in the appendices, and GroundFscore uses 100 held-out observations; shortcut probes use separate lists of 50 trials per policy.

Why This Matters

Impact on research. The paper reframes data scaling for robot learning: instead of asking only whether more data raises success, it asks whether the data demand the right decisions. GroundFscore offers a cheap offline alternative to costly shortcut-targeted rollouts, and the three interventions are deliberately designed to be composable with existing VLA training rather than replacing it.

Real-world applications:

• Warehouse and logistics picking, where instructions name a specific object among clutter and a policy must not reach for whatever is most visually salient.
• Household service robots, where a user may say "push" rather than "pick up" and the policy must ground the verb rather than replaying a familiar grasp.
• Industrial assembly, where a gripper closing on an empty grasp and lifting anyway could damage parts or the tooling.
• Dexterous and bimanual manipulation, where RoboTwin 2.0 tasks such as opening a microwave, hammering, and hanging a mug require outcome-dependent behavior rather than scripted sequences.

Industry relevance. General-purpose robot foundation models are increasingly trained on large heterogeneous demonstration corpora. If those corpora preserve the same correlations, scale buys proficiency where shortcut and evidence agree but leaves brittleness where they disagree. PerturbBot's claim — that it changes what is scaled without changing inference — targets exactly this bottleneck, and its auxiliary data source (existing random-motion and failed-execution recordings) is something real deployments already generate.

Future Directions

• Extending beyond recorded behavior. The authors state that relabeling cannot create behaviors never recorded, such as recovery after an empty grasp; a natural next step is collecting and incorporating corrective continuations evaluated under the original instruction.
• Reducing annotation dependence. PerturbBot currently requires detailed captions and relabeled segments matched to recorded behavior, plus random and failed trajectories beyond demonstrations. Lowering that annotation cost would broaden applicability.
• Testing whether GroundFscore holds under broader distribution shift and across architectures. The validity test covers 36 checkpoints from twelve conditions and three seeds on General PnP; the RoboTwin verb-change probes for noun lock-in sit in the appendix, and whether the score's ranking advantage transfers to other backbones is not established.
• Determining whether the coverage added by R translates into genuinely new skills. The paper frames R as widening language-conditioned state-action coverage without teaching recovery, leaving open how much of the gain comes from coverage versus from breaking degeneracy.

Target Audience

Robotics and embodied-AI researchers working on VLA policies, imitation learning, and data-centric training; practitioners who need to diagnose whether a deployed manipulation policy is using task evidence or replaying a cue; and evaluation-focused researchers interested in offline metrics that substitute for expensive robot rollouts. Readers without a background in multimodal policy learning or information-theoretic decomposition will find the method sections challenging, but the failure-mode framing and the empirical tables are accessible to a broader engineering audience.

Note: the truncated content does not report the paper's funding sources, the exact number of training demonstrations for RoboTwin 2.0, the wall-clock cost of caption generation, or the full appendix tables referenced as Tables 7, 9, 10, and 13.

Authors’ abstract

A vision--language--action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target. Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway. We call these dependencies modality shortcuts: regularities in successful demonstrations make visual, lexical, or motor cues sufficient to predict expert actions without the task evidence needed for the underlying decision. More demonstrations of the same kind can raise task success while leaving these shortcuts intact. We propose Perturbot which makes task-relevant evidence easier to use and shortcuts insufficient on their own: it applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. It complements scaling by changing what is scaled, and leaves inference unchanged. Moreover, we propose GroundingFscore, an offline score that diagnoses how severely a policy relies on modality shortcuts. Task success rate shows whether a policy improves, while GroundingFscore reveals whether the policy scales healthily, relying on task evidence rather than shortcuts. Together, Perturbot and GroundingFscore provide a training-and-evaluation framework for disentangling VLA decisions from shortcut priors while preserving responsiveness to task-relevant evidence.