TacitVLA: Reasoning Vision-Language-Action Models Without Reasoning Latency

AI summary

Overview

Research area: Robotics and embodied AI — vision-language-action (VLA) models, reasoning-before-acting policies, and inference latency for real-time control.

Technical level: Advanced. The paper assumes familiarity with vision-language model backbones, transformer attention masking, Mixture-of-Transformers coupling, flow-matching action experts, and benchmark protocols for manipulation and autonomous driving.

Scope in one sentence: A controlled study showing that a VLA action expert never needs to condition on the reasoning text generated for the current call, allowing the same architecture to keep reasoning's performance benefits while removing reasoning's latency from the action path.

What This Paper Is About

Reasoning VLAs generate text — a scene description, a plan, or the next subtask — before producing an action, and every action must wait for that autoregressive decoding to finish. The authors ask whether conditioning the action on the current reasoning text is actually necessary, or whether simply training the model to generate reasoning is enough to get its benefits. Their answer is that the supervision alone improves the backbone representation the action expert already conditions on, so the generated text can be skipped at deployment on non-memory tasks, and generated during action execution on tasks where the text also serves as memory.

Key Contributions

1. A controlled comparison isolating one variable. The authors train Explicit Reasoning and TacitVLA on identical sequences, data, prompts, and optimizers, differing only in whether the action expert's attention is masked from the reasoning tokens (plus No Reasoning, trained without reasoning supervision). This isolates the contribution of attending to reasoning text from the contribution of reasoning supervision.

2. TacitVLA: Explicit Reasoning's success at No Reasoning's latency. Across all four benchmarks, TacitVLA matches Explicit Reasoning's performance while running at up to 1.7x lower latency on memory tasks and up to 3.4x lower latency on non-memory tasks (Section 6.2), and while matching No Reasoning's latency it improves success by up to +17.1% (Section 6.1).

3. Asynchronous ledger generation for memory tasks. On memory tasks where each generated subtask becomes the next entry in a ledger, TacitVLA generates the entry during action execution rather than before it, so the next call never waits for generation (Equation 4).

4. State-of-the-art absolute results from a single backbone feeding one action expert. TacitVLA surpasses the best previously published results on the RoboMME Counting suite (82.7% vs. 66.8%) and RMBench (77.5% vs. 56.5%), and reaches 72.9% on LIBERO-Plus and a 52.5% scene score on the public split of the PhysicalAI-AV closed-loop driving challenge — with no second model or scheduler added.

Main Findings

• Reasoning text is a large share of latency. For the reasoning VLAs the authors train on LIBERO and PhysicalAI-AV, generating the reasoning text accounts for 53% to 70% of the latency per call (Section 6.2).

• Matching Explicit Reasoning on RoboMME Counting. Trained on the Counting suite alone and evaluated on its four tasks (200 cases), TacitVLA scores 89.5 ± 1.3, Explicit Reasoning 86.7 ± 2.0, and No Reasoning 79.0 ± 1.0, against a human reference of 88.5. In that human reference, people chose each high-level action and an oracle planner executed it without error, whereas TacitVLA must also execute every motion.

• Matching or exceeding the best published result on RoboMME. Trained on all 16 tasks (800 cases), No Reasoning, Explicit Reasoning, and TacitVLA score 46.5, 45.8, and 48.1 on all tasks, compared with 44.5 for FrameSamp + Modulator, the benchmark's best published result. On the counting tasks, TacitVLA's 82.7 is within 1.2 points of the oracle-subgoal rows (83.9 and 82.6) and above trained symbolic systems (38.0 to 48.8).

• Memory is what drives the RMBench gap. The three policies are within a point of one another on Blocks Ranking Try, Battery Try, and Cover Blocks. On Press Button, whose count exists only in the ledger, Explicit Reasoning and TacitVLA score 73.3 and 74.3 against 7.7 for No Reasoning. Four-task means are 60.4, 77.0, and 77.5 for No Reasoning, Explicit Reasoning, and TacitVLA, all above WLA-0's 56.5.

• Ledger accuracy is preserved. Both reasoning policies keep a correct ledger in 99.7% of RMBench episodes, and TacitVLA's final ledger matches the instructed count within two points of Explicit Reasoning's on each counting task.

• Reasoning supervision, not text conditioning, lifts LIBERO-Plus. On LIBERO-Plus, No Reasoning scores 67.6, Explicit Reasoning 73.5, and TacitVLA 72.9. On the four standard suites the three score 98.5, 98.1, and 98.6. The gain over No Reasoning is largest on changed robot initial states, 11.4 points for Explicit Reasoning and 10.2 for TacitVLA. TacitVLA generates no text on these tasks, so the lift reaches its action through the representation alone.

• Driving results. In open loop, both reasoning policies lower displacement error below No Reasoning's 1.166 m, to 1.140 m for Explicit Reasoning and 1.116 m for TacitVLA. In closed loop on the 441-scene public split, TacitVLA reaches a 52.5% scene score and Explicit Reasoning 52.3%, against 46.4% for No Reasoning; both reasoning policies pass more scenes and cause fewer at-fault collisions. Alpamayo-R1, a 10B model with four cameras, scores 55.0% on the same scenes and protocol, and VaVAM scores 11.0% under the competition controller and 31.4% under a linear controller not available in the competition.

• Where the latency comes from, per benchmark. On RoboMME counting, the action path takes T_action = 127 ms and generating the ledger entry takes T_reason = 93 ms; TacitVLA generates during execution in 12% of the 800 ms window (16 actions at 20 Hz), so its action is ready after 127 ms, while Explicit Reasoning's is ready after 220 ms, 1.7x later. On RMBench, generation takes 66 to 120 ms, at most 1.1% of the 7.0 to 11.2 s window; TacitVLA's action is ready after 169 to 171 ms and Explicit Reasoning's after 236 to 294 ms, 1.4x to 1.7x later. On non-memory tasks, TacitVLA runs at 147 ms per call on LIBERO and LIBERO-Plus and 127 ms in closed-loop driving, versus 495 ms and 269 ms for Explicit Reasoning, 3.4x and 2.1x later. LIBERO's 250 ms window (5 actions at 20 Hz) is met by TacitVLA and not by Explicit Reasoning.

• Fixing one attention-mask cell is the whole difference. TactVLA uses slightly less training compute per update because its action positions do not attend to the reasoning tokens.

Methodology in Plain English

The authors take the architecture recent reasoning VLAs adopt — a vision-language backbone that generates reasoning text feeding a flow-matching action expert — and train three versions of it on exactly the same data, sequences, prompts, loss, and optimizer.

• No Reasoning is trained without reasoning supervision. It generates no text and sets the latency floor.
• Explicit Reasoning deploys as embodied chain-of-thought does: the action expert attends to the reasoning text generated for the current call, so every action chunk waits for decoding to finish.
• TacitVLA is trained on the same sequence but with the action expert's attention to the reasoning tokens masked out. It still learns to generate reasoning, but its actions never depend on that text.

This turns the question into a clean experiment. Comparing TacitVLA with No Reasoning isolates the benefit of reasoning as a training target; comparing it with Explicit Reasoning isolates the benefit of conditioning on the generated text. The authors run this comparison on four benchmarks split by how the reasoning text is used. On memory tasks (RoboMME, RMBench), each generated subtask is appended to a ledger that later calls condition on, so the text must still be produced — TacitVLA generates it during the execution window, after the action is computed, and delays the next call only if generation outlasts execution. On non-memory tasks (LIBERO, PhysicalAI-AV), the text serves only the current action and is discarded, so TacitVLA does not generate it at all. Latency is measured with per-call timers during evaluation episodes, and all three policies run the same inference implementation on one H100.

Why This Matters

The paper removes what has been a structural trade-off in embodied AI: reasoning improves policies but its autoregressive decoding sits directly on the action path. Prior work either accepted the wait, ran the language model at a lower rate than a separate controller, or switched generation off on a checkpoint trained to condition on it. TacitVLA shows that the performance benefit of reasoning survives when the action never waits for the text, and it extends that result to memory tasks where the reasoning text cannot simply be discarded — an axis earlier work did not address. The change is a single attention mask, so any reasoning VLA of this form can adopt it, and the latency is measurably reduced rather than traded away.

Real-world applications:

• Robot manipulation: household and industrial arms running long-horizon tasks where a control loop window (for example LIBERO's 250 ms) rules out on-path text generation.
• Autonomous driving: closed-loop planning where per-call latency budgets are tight, with the deferred reasoning text available as an interpretability artifact for safety-critical review.
• Memory-dependent task execution: procedures such as counting repeated button presses or tracking which subtask has already been attempted, where the ledger must persist across calls.
• Deployment on fixed-rate hardware: any system whose controller runs at a fixed frequency and cannot stall for text decoding.

Industry relevance: the work comes from NVIDIA with collaborators at NYU, UC Berkeley, Johns Hopkins, and Stanford, and is evaluated on NVIDIA's PhysicalAI-AV and AlpaSim driving stack alongside standard robotics benchmarks. Its practical appeal is that it requires no additional model, no scheduler, and no change to the training pipeline beyond masking — an incremental change to an existing recipe rather than a new system.

Future Directions

• A three-expert backbone. Fast-WAM reaches an analogous conclusion for future video prediction. Combining the two suggests a backbone with three experts for reasoning, future video, and action, trained on all three targets but acting without first generating reasoning or future video.

• Deferred reasoning as an interpretability channel. On non-memory tasks, TacitVLA can still generate reasoning text during action execution. The authors propose this as a way to surface explanations in safety-critical settings such as autonomous driving, and leave it to future work.

• Robustness to the execution-window budget. The deferred entry adds no latency only while it fits inside the execution window. It takes at most 23% of the window on these benchmarks; longer entries or shorter windows add the time by which generation exceeds the window, per Equation 4.

• Dependence on reasoning data quality. As with other reasoning VLAs, performance depends on the quality of the reasoning data, and the paper's recipe inherits that dependency.

Target Audience

Researchers and engineers working on vision-language-action models, embodied chain-of-thought, and real-time robot or vehicle control — particularly those who need reasoning's generalization benefits but cannot afford its decoding cost. It is also relevant to practitioners building inference systems for robotics, who will find the per-stage latency breakdown and the ledger-generation scheduling discussion directly useful, and to readers following the broader question of when chain-of-thought must be generated versus only trained on.

Authors’ abstract

Textual reasoning before acting has improved vision-language-action (VLA) models, but at the cost of latency: every action conditioned on the reasoning text must wait for autoregressive decoding. We show that actions do not need to wait for or condition on the generated reasoning to benefit from it, because the reasoning supervision already improves the backbone representation. We introduce TacitVLA, a reasoning VLA whose action expert never attends to the reasoning it generates for the current call. On long-horizon tasks where the reasoning text also serves as episodic memory, it can generate each entry during action execution rather than before acting. We evaluate it in a controlled study on four benchmarks spanning manipulation and autonomous driving, memory and non-memory tasks: RoboMME, RMBench, LIBERO, and PhysicalAI-AV. On all four, TacitVLA matches the performance of its explicit-reasoning counterpart at up to 3.4x lower latency, and matches the latency of the same architecture trained without reasoning at up to +17.1% success. In absolute terms, it surpasses the best previously published results on the RoboMME Counting suite (82.7% vs. 66.8%) and RMBench (77.5% vs. 56.5%), and reaches 72.9% on LIBERO-Plus and a 52.5% scene score on the public split of the PhysicalAI-AV closed-loop driving challenge. Project page: https://tacitvla.github.io.