Say the Same, Act Differently: Diagnosing and Steering Text-Action Divergence in Reasoning VLAs

AI summary

Overview

Research area: AI safety and adversarial robustness for reasoning vision-language-action (VLA) models, spanning autonomous driving and robotic manipulation.

Technical level: Advanced. The paper relies on Jacobian-based directions in hidden-state space, gradient projection, and projected-gradient-descent attacks, though the core argument is stated plainly enough to follow without reproducing the math.

Scope: The paper diagnoses a geometric decoupling between the text channel and the action channel in two reasoning VLAs (Alpamayo for driving, InstructVLA for manipulation), and shows that image perturbations guided along that decoupling can change physical actions while leaving the generated rationale word-for-word identical.

What This Paper Is About

Reasoning VLAs produce a natural-language rationale alongside their actions, and supervisors are meant to read that rationale to catch unsafe behavior before it happens. Prior attacks on these models corrupted the reasoning text itself, so the tampering was visible in the channel a monitor reads. This paper asks the converse question: can the physical action become unsafe while the generated rationale stays exactly the same, and if so, what does that imply for anyone relying on the text as a safety certificate?

Key Contributions

1. Rationale-preserving action divergence through visual inputs. The authors demonstrate that visual perturbations alone can substantially alter a reasoning VLA's action while preserving the exact generated rationale. In Alpamayo this produces a new collision under a word-for-word identical rationale in 52.6% of 154 obstacle-annotated scenes under full-image perturbations and 35.1% under a localized forward-camera patch, versus 4.55% for unguided action attacks. In InstructVLA, 85.0% of examples keep the exact response alongside a 1.73-unit action-chunk shift.

2. A local geometric mechanism for text–action decoupling (TNAS). They introduce the text-neutral action-subspace diagnostic, which projects the locally most action-sensitive hidden-state direction away from the active text-loss span. The resulting action direction overlaps that text span by only 1.87% of its norm in Alpamayo and 5.28% in InstructVLA, meaning projecting out the text span retains more than 99.8% of the action direction's norm on average.

3. Hidden-state causal controls. Under equal perturbation norms, steps along the text-neutral direction shift actions 4.64× to 47.8× more than text-gradient perturbations, while text-gradient perturbations induce 21.2× to 277× more teacher-forced KL divergence.

4. Mechanism-guided attacks that resolve an optimization conflict. Naively combining an action-deviation loss with an output-level text penalty creates competing gradients. Adding TNAS alignment raises exact rationale preservation from 83.0% to 89.5% in Alpamayo and from 77.5% to 85.0% in InstructVLA, while teacher-forced text drift falls by 68.4% and 40.5% respectively.

Main Findings

• The rationale is informative but not binding. A frozen Qwen3-14B reader predicting from the clean rationale alone matched Alpamayo's stop-versus-continue trajectory outcome on 84.3% of 1,181 clips, where 24.5% (289/1,181) of plans actually come to rest. Clips predicted to stop came to rest 63.6% of the time versus 5.76% for clips predicted to continue.

• Identical text coexists with very different actions. Those 1,181 clips share only 258 distinct rationales, and 36.9% carry a rationale that also appears with the opposite stopping outcome. The single most frequent rationale ("Keep distance to the lead vehicle since it is directly ahead in our lane") covers 245 clips with 10.1 m to 208 m of forward displacement.

• The action-sensitive direction is nearly orthogonal to the text span. The projection-norm ratio of the action direction onto the text-loss span averages 1.87% in Alpamayo and 5.28% in InstructVLA, against roughly 0.978% for isotropic random unit vectors. The text-loss span has rank 2 on every evaluated example.

• Finite hidden-state perturbations confirm the local geometry. At a fixed teacher-forced KL budget of 10⁻⁴ and 5×10⁻⁴, the text-neutral direction shifted Alpamayo's plan by 0.227 m and 0.504 m, exceeding the controls (≤5.10 mm) by 63.3× to 212×. In InstructVLA at a budget of 10⁻³, it reached 1.37×10⁻² and 1.66×10⁻² versus at most 9.18×10⁻³ for the controls.

• Equal-norm probes show intrinsic selectivity, not just a larger step. In Alpamayo at q=2 the text-neutral direction shifted the trajectory 0.142 m, exceeding text-gradient controls by 44.0× to 47.8× and the random control by 53.9×, while inducing only 9.87×10⁻⁵ teacher-forced KL against 269× to 277× higher KL for the text-gradient controls. In InstructVLA at q=2 it shifted the action by 0.156, exceeding text-gradient controls by 4.64× to 5.51× and the random control by 12.6×, with KL of 2.20×10⁻² versus 21.2× to 26.3× higher for the text-gradient controls.

• Output-level text penalties fight the action objective. In Alpamayo, Action-only PGD shifted the plan by 70.1 m ADE but rewrote the rationale on 90.5% of clips, leaving only 4.55% silent collisions. Adding the text penalty (Text-preserving PGD) kept the rationale intact on 83.0% of clips and raised silent collisions to 48.7%, but pushed vocabulary-wide teacher-forced KL from 2.78×10⁻¹ to 3.26×10⁻¹, making stochastic reproduction of the clean rationale less likely.

• Steering along the text-neutral direction quiets the text channel. Increasing the alignment weight from 5 to 30 in Alpamayo raised cosine alignment from 1.64×10⁻² to 2.46×10⁻¹ (versus 8.00×10⁻⁴ under Text-preserving PGD) and pulled teacher-forced KL down monotonically to 1.03×10⁻¹, a 68.4% reduction; exact free-generation match rose monotonically to 89.5%, and silent collisions reached 52.6%.

• Displacement is traded for concentration, not lost. Fixed-prefix displacement fell from 69.2 m to 53.0 m at the strongest guidance, yet free-generated plans collided on 61.7% of the 154 annotated clips at the strongest weight versus 61.0% under Text-preserving PGD.

• The geometry transfers to manipulation. In InstructVLA, cosine alignment rose from 1.22×10⁻¹ to 2.13×10⁻¹ across the sweep, teacher-forced KL fell from 2.67×10⁻² to 2.06×10⁻² (40.5% below Text-preserving PGD and 55.8% below Action-only PGD), and exact match rose from 81.0% to 85.0%, while retaining 87.5% of Text-preserving PGD's action displacement.

• Localized patches retain the effect. A shared forward-camera patch on 13.1% of visual tokens in Alpamayo cut teacher-forced KL by 59.0% across the sweep, preserved up to 80.5% of rationales word for word, and induced silent collisions on up to 35.1% of annotated scenes. In InstructVLA, a 0.4×0.4 rectangle covering 16% of the image produced a 1.32-unit action shift, matching the 77.5% exact preservation of full-image Text-preserving PGD with 19.1% lower teacher-forced KL.

Methodology in Plain English

The authors treat the model as having a shared internal state that both the text and the action read out from. Because turning that continuous state into discrete words is a many-to-one mapping, many different states can produce the exact same sentence.

1. Find the action-sensitive direction. They locate the internal direction that the predicted action is locally most sensitive to (the "action-Jacobian direction"). They deliberately do not differentiate the action-shift magnitude itself, because that quantity is zero at the clean input and has no well-defined gradient there.

2. Find the text-sensitive directions. Two losses that are still active at the clean point are used: the target token's negative log-likelihood and the drop in the logit margin against the strongest competing token. Teacher-forced KL and hinge losses are excluded because their gradients vanish at the clean input. These two gradients form a rank-2 span.

3. Project the text out. Subtracting the text-span component from the action direction yields the text-neutral action-subspace direction. By construction, a small step along it leaves the rationale's likelihood and margin unchanged to first order while still moving the action.

4. Test on the hidden state, not just the gradient. Two protocols are run: one matches a fixed text-drift budget across directions, and one applies an identical perturbation magnitude across directions. Both compare the text-neutral direction against the raw action direction, the two text-gradient directions, and random unit vectors. Subsequent probes use finite, nonlinear perturbations through the language and action heads.

5. Carry it back to the image. The attack optimizes an image perturbation with 200 steps of projected gradient descent, using an action-displacement loss plus an explicit text-preservation penalty plus a cosine-alignment term that steers the induced internal shift toward the text-neutral direction. This replaces the conflict between the action and text objectives with a single direction that serves both.

6. Score it end to end. Evaluation covers fixed-prefix readouts (action shift, teacher-forced KL, alignment), free-generation word-for-word exact match, and, for driving, an open-loop collision check that projects the ego vehicle and logged actors as oriented bounding boxes in bird's-eye view over the 6.4 s horizon. The "silent collision rate" counts clips where a collision-free clean plan becomes a collision while the rationale stays identical.

The models and cohorts are Alpamayo on 200 clips and InstructVLA on 200 Bridge-distribution examples, with 154 Alpamayo clips carrying obstacle annotations. The hooked layers are reasoning-token layers 32–35 in Alpamayo and response-token layers 14–17 in InstructVLA. Action readouts use a fixed 10-step Euler flow-matching solver with paired initial noise, over 64 waypoints at 10 Hz in Alpamayo and 16 chunk steps of 7-DoF end-effector actions in InstructVLA.

Why This Matters

Impact on research. The paper reframes the safety case for reasoning models in embodied settings. Prior work showed that corrupting the reasoning trace changes behavior, which implies a monitor reading the trace can catch it. This work shows the converse failure: the trace can be perfectly preserved while behavior changes, so a rationale monitor is not a certificate of action safety. It also supplies a mechanistic explanation, grounded in the geometry of the shared interface, for why output-level text penalties and action objectives interfere during optimization.

Real-world applications:

• Runtime safety monitors for autonomous driving. A monitor that only checks the generated rationale ("Stop to keep distance to the lead vehicle") would pass a plan that continues into the lead vehicle. The result argues for monitors that verify the predicted action channel directly.

• Robotic manipulation safety. In warehouse or home manipulation, a concise task-intention sentence ("I will move the cup to the sink") may remain unchanged while the 16-step action chunk shifts, so a language-level check would miss the deviation.

• Adversarial evaluation and red-teaming pipelines. The text-neutral direction gives a concrete way to stress-test a VLA's text–action consistency before deployment, rather than relying on generic perturbation baselines.

• Localized physical-world threats. The patch results matter for scenarios where a perturbation is attached to a local object or surface, such as a patch on the back of a front vehicle or a printed region on a tabletop.

Industry relevance. Any organization deploying VLA-style systems where a human is expected to supervise model reasoning, including automotive, robotics, and logistics, has a direct interest in the finding that an unchanged explanation cannot certify a safe action. The technique is white-box and requires access to model internals, but the safety conclusion applies to black-box monitoring practice as well.

Future Directions

• Action-channel monitors. Since the rationale cannot certify safety, what does a practical runtime monitor look like that verifies the predicted action directly, and at what latency and cost?

• Generalization across models and embodiments. Only two models (Alpamayo and InstructVLA) are studied, with layers selected by an explicit qualifying-block criterion. Whether the same low overlap between the action direction and the text span holds in other reasoning VLA architectures, and in models with different text readouts, is an open question.

• Defenses at the shared interface. The text-neutral direction is identified in a white-box setting. Whether training-time regularization, architectural separation of the text and action readouts, or detection of internal shifts along this direction can close the gap is unexplored.

• Physical-world validation of localized patches. The patch experiments are digital perturbations on the processed input tensor, with a larger local budget (ε=0.5) to compensate for restricted spatial support. Whether comparable silent divergence survives real-world capture, viewpoint change, and printing is not established here.

Target Audience

Researchers and engineers working on embodied AI safety, VLA and robot-learning systems, and adversarial robustness, particularly those designing supervision or monitoring interfaces around model-generated reasoning. It is also relevant to safety reviewers and certification teams in automotive and robotics who currently treat natural-language rationales as an auditable signal. The paper assumes familiarity with transformer internals, gradient-based attacks, and action-prediction heads, so readers without that background will find the geometry sections demanding, though the headline failure mode is stated clearly enough for a policy or safety-audit audience to absorb.

Authors’ abstract

Reasoning vision-language-action (VLA) models pair physical control with natural-language rationales that supervisors can read to monitor embodied safety. Prior attacks on VLA reasoning corrupt the rationale itself and leave visible evidence in the text. We show that visual perturbations can instead alter the physical action while keeping the generated rationale word-for-word identical. At the shared text/action interface of Alpamayo for driving and InstructVLA for manipulation, our text-neutral action-subspace (TNAS) diagnostic shows that the locally most action-sensitive direction is nearly orthogonal to the active text-loss gradients, with 1.87% and 5.28% norm overlap. At equal hidden-state perturbation norms across both models, interventions along this text-neutral residual shift actions 4.64x to 47.8x more than text-gradient perturbations, whereas text-gradient perturbations induce 21.2x to 277x more text drift. Steering visual perturbations along this direction reaches the same decoupling from the image. Across 154 obstacle-annotated driving scenes in Alpamayo, TNAS-guided attacks induce a new collision under a word-for-word identical rationale in 52.6% of scenes under full-image perturbations and 35.1% under a localized forward-camera patch, compared with 4.55% for unguided action attacks. Relative to a text-preserving baseline, TNAS guidance raises exact rationale preservation from 83.0% to 89.5% and cuts teacher-forced text drift by 68.4% at a comparable collision rate. In InstructVLA, TNAS guidance cuts text drift by 40.5% and preserves the exact response on 85.0% of examples alongside a 1.73-unit action-chunk shift. An unchanged rationale therefore cannot certify a safe action, so monitors for reasoning VLAs must verify the predicted action channel directly.