Latent Reasoning VLA: Latent Thinking and Prediction for Vision-Language-Action Models

AI summary

Overview

• Research area: Robotics, specifically Vision-Language-Action (VLA) models that combine vision, language and control for robotic manipulation. The paper sits at the intersection of embodied AI, chain-of-thought (CoT) reasoning, and latent-space representation learning.
• Technical level: Advanced. The paper assumes familiarity with VLA architectures, autoregressive decoding, diffusion/flow-matching action heads, and latent reasoning methods from language models.
• Scope: In one sentence — the paper introduces LaRA-VLA, a framework that moves both textual and visual chain-of-thought reasoning into continuous latent representations so that a VLA model can reason and act without generating explicit reasoning tokens at inference time.

What This Paper Is About

Existing VLA models benefit from chain-of-thought reasoning, but generating that reasoning as text or as discrete visual tokens is slow at inference and forces reasoning into a discrete format that does not match the continuous perception and action spaces of robotics. The authors propose LaRA-VLA, which internalizes multi-modal chain-of-thought into continuous latent "thought" representations through a three-stage curriculum, so that reasoning and prediction happen entirely in latent space and actions are produced without explicit CoT generation. The goal is to keep the benefits of structured intermediate reasoning while making control fast enough for real-time robotic deployment.

Key Contributions

1. A latent-reasoning paradigm for VLA models. Chain-of-thought reasoning is internalized into continuous latent representations across both textual and visual modalities, enabling inference-efficient reasoning that aligns with continuous perception and control.
2. The LaRA-VLA model and its curriculum training strategy. A unified VLA model that progressively transitions from explicit multi-modal CoT supervision to latent embodied reasoning, guided by predictive visual latent objectives and stabilized with EMA-based visual encoders.
3. Two structured chain-of-thought datasets. LIBERO-LaRA and Bridge-LaRA, built with an automated annotation pipeline that provides multi-modal reasoning annotations (subtask decomposition, movement reasoning, target object localization) for embodied manipulation.
4. Extensive evaluation in simulation and on real hardware. Experiments on LIBERO and SimplerEnv plus long-horizon real-robot manipulation tasks, showing consistent gains over state-of-the-art VLA methods and up to a 90% reduction in inference latency relative to explicit CoT approaches.

Main Findings

• LIBERO results: LaRA-VLA reaches an average success rate of 97.9%, with 96.4 on Spatial, 98.6 on Goal, 99.8 on Object, and 96.6 on Long. The paper states this is the best overall performance among compared methods, including No-CoT baselines (OpenVLA, pi_0, OpenVLA-OFT), textual CoT baselines (ThinkAct, MolmoAct, pi_0.5, DeepThinkVLA), visual CoT baselines (CoT-VLA, DreamVLA, F1, UD-VLA), and the latent CoT method Fast-ThinkAct.
• SimplerEnv results: LaRA-VLA attains the highest average success rate of 68.8%, with 95.8 on Put Spoon, 62.5 on Put Carrot, 25.0 on Stack Block, and 91.7 on Put Eggplant. It outperforms No-CoT, Textual CoT and Visual CoT baselines on average.
• Real-world long-horizon manipulation: On an Agilex Cobot Magic wheeled platform with three RGB-D cameras, LaRA-VLA achieves the highest average success rate among all compared methods, substantially outperforming ACT and ECoT and surpassing GR00T N1.5 overall. It attains the best performance on three out of four long-horizon tasks, with notable gains on tasks requiring multi-stage reasoning and sustained temporal coordination.
• Ablation of CoT supervision (SimplerEnv success rate): No CoT 55.2; explicit Text-CoT 58.3; Latent Text-CoT 64.6; Latent Vis-CoT 63.5; Latent Text-CoT plus Latent Vis-CoT 68.8. Explicit textual CoT gives only marginal improvement over the no-CoT baseline, while latent CoT gives substantially larger gains.
• Inference efficiency: LaRA-VLA requires 135 ms per rollout on an NVIDIA A100 GPU, the lowest latency among all compared methods. It avoids autoregressive textual reasoning and reduces inference time by up to 90% compared to explicit CoT methods, and is also faster than Fast-ThinkAct, which also uses latent CoT reasoning.
• Robustness under visual corruption: Under Gaussian blur and Gaussian noise at two severity levels, LaRA-VLA maintains higher success rates than the no-CoT latent-free baseline Qwen-GR00T across all perturbation types. On LIBERO: blur-H 42.9 vs 30.0, blur-L 79.4 vs 76.0, noise-H 76.0 vs 55.7, noise-L 92.7 vs 87.9. On SimplerEnv: blur-H 56.3 vs 13.5, blur-L 62.5 vs 35.4, noise-H 22.9 vs 8.3, noise-L 31.2 vs 22.9.
• No latent collapse observed: Latent tokens for different reasoning components form well-separated, semantically coherent clusters with clear functional specialization, and language instruction token latents occupy a distinct subspace from reasoning latents, indicating latent CoT does not trivially reuse language embeddings.
• Motivation from prior CoT costs: The paper states that text-based CoT models may operate at control frequencies below 5 Hz, or even around 1 Hz, which is unacceptable for real-time robotic control.

Methodology in Plain English

The authors start by building structured supervision automatically. An annotation pipeline uses semantic anchors (the manipulated object, extracted with Qwen3-VL from the initial frame and instruction) and temporal anchors (trajectories segmented into atomic manipulation stages based on gripper state changes). Conditioned on these anchors, Qwen3-VL generates concise subtask descriptions, while GroundingDINO and SAM3 produce temporally consistent target-object bounding boxes. Motion reasoning is derived from end-effector trajectories by computing goal-directed and local motions, discretized into directional descriptors. This pipeline produces the LIBERO-LaRA and Bridge-LaRA datasets from LIBERO and SimplerEnv, and is also applied to real-world robot data.

The model uses Qwen3-VL as the backbone VLM and inherits its image encoder for consistent visual representations. A dedicated <img_next> token represents predicted visual latents; in the final stage a separate action expert — a 16-layer Diffusion Transformer with alternating self-attention and cross-attention layers — produces continuous action trajectories.

Training has three stages. In Stage I, the model is fine-tuned with explicit CoT annotations using teacher forcing, jointly optimizing a CoT negative log-likelihood loss, a visual latent alignment loss (L1 between predicted and encoded next-observation latents, weight 0.1), and a discrete autoregressive action-token loss. Target visual latents come from an EMA-updated copy of the visual encoder with decay rate tau_v, to stabilize learning and prevent representation collapse. In Stage II, explicit CoT tokens are progressively masked and replaced by learnable latent representations according to a curriculum schedule; the CoT supervision loss is annealed to zero and the model is finally optimized with 0.2 times the visual latent loss plus the discrete action loss. In Stage III, discrete action prediction is dropped and the action expert is trained with a flow-matching objective, where a velocity field is regressed conditioned on a multi-modal latent context that aggregates the current visual observation and instruction, the text reasoning latent, and the predicted future visual latent. A tailored attention mask controls information flow among text, current image, future image and action tokens at each stage.

Why This Matters

• Impact on research: The paper argues that CoT is effective because it exposes structured intermediate reasoning, not because it is written in natural language. It shows that this reasoning can be internalized in continuous latent space for embodied agents, and it extends latent CoT work from language-only and vision-language settings to VLA control. It also provides a taxonomy (Table 1) distinguishing discrete versus continuous text CoT, the visual alignment form (raw pixels, discrete tokens, or continuous latents), and discrete versus continuous actions.
• Real-world applications (as implied by the evaluation setups):
  • Real-time robotic manipulation on mobile wheeled platforms with multiple RGB-D cameras.
  • Long-horizon household-style tasks: putting objects into a basket, sorting fruits, finding and placing a block, and stacking bowls.
  • Tabletop manipulation with WidowX-style arms, tested for real-to-sim generalization via SimplerEnv.
  • Deployment scenarios where perception is noisy or corrupted, since the method preserves higher success rates under Gaussian noise and blur.
• Industry relevance: Inference latency of 135 ms per rollout and latency reductions of up to 90% relative to explicit CoT are directly relevant to control loops, since the paper notes explicit text CoT can run below 5 Hz or near 1 Hz. A design that avoids long autoregressive reasoning tokens also reduces KV-cache usage, memory consumption and token expansion, which matters for deploying VLAs on real robotic hardware.

Future Directions

• Scaling latent reasoning capacity: The authors restrict latent reasoning to a single token per step to avoid latent collapse as the number of latent tokens grows; relaxing this limit while preventing collapse is presented as an open problem.
• Training efficiency: The curriculum replaces explicit CoT tokens with latents gradually, and the number of CoT-related tokens increases as training progresses, raising training cost. Improving training efficiency while preserving stable latent reasoning is stated as an important direction.
• Managing the collapse-versus-expressiveness trade-off: The paper cites prior work showing latent token semantics can degenerate toward homogeneous representations as the number of latent tokens increases, so supervision strategies that keep latents structured at larger scale remain unresolved.
• Broadening evaluation: The current evidence covers LIBERO, SimplerEnv and a real-world dataset collected on the authors' platform; whether the same latent reasoning dynamics transfer to other robot morphologies, tasks and annotation pipelines is not reported.

Target Audience

This paper is most useful for robotics and embodied-AI researchers working on Vision-Language-Action models, chain-of-thought reasoning, or efficient inference for control policies. It also suits machine learning engineers interested in latent reasoning methods transferred from language and vision-language models to continuous control, and practitioners evaluating whether reasoning-augmented VLA models can meet real-time control constraints. Readers should already be comfortable with transformer attention, autoregressive decoding, diffusion or flow-matching action heads, and robot manipulation benchmarks.

Authors’ abstract

Vision-Language-Action (VLA) models benefit from chain-of-thought (CoT) reasoning, but existing approaches incur high inference overhead and rely on discrete reasoning representations that mismatch continuous perception and control. We propose Latent Reasoning VLA (LaRA-VLA), a unified VLA framework that internalizes multi-modal CoT reasoning into continuous latent representations for embodied action. LaRA-VLA performs unified reasoning and prediction in latent space, eliminating explicit CoT generation at inference time and enabling efficient, action-oriented control. To realize latent embodied reasoning, we introduce a curriculum-based training paradigm that progressively transitions from explicit textual and visual CoT supervision to latent reasoning, and finally adapts latent reasoning dynamics to condition action generation. We construct two structured CoT datasets and evaluate LaRA-VLA on both simulation benchmarks and long-horizon real-robot manipulation tasks. Experimental results show that LaRA-VLA consistently outperforms state-of-the-art VLA methods while reducing inference latency by up to 90\% compared to explicit CoT-based approaches, demonstrating latent reasoning as an effective and efficient paradigm for real-time embodied control. Project Page: https://loveju1y.github.io/Latent-Reasoning-VLA/