LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries

AI summary

Overview

Research area: Robot learning / Vision-Language-Action (VLA) models — specifically, how to make a robot policy actually obey the language instruction rather than guessing the task from the scene.

Technical level: Advanced (assumes familiarity with Bayesian policy factorization, mutual information, flow-matching diffusion policies, and transformer causal masking).

Scope: The paper diagnoses a dataset-driven failure mode called "Information Collapse" — where VLA models ignore language because vision alone predicts the instruction — and proposes LangForce, a dual-branch training framework using Latent Action Queries to maximize the pointwise mutual information between actions and instructions.

What This Paper Is About

Most robot demonstration datasets are collected in a goal-driven way (the same person doing the same task in the same scene), which makes the language instruction nearly predictable from the camera image alone. The authors show this determinism lets a VLA model succeed without ever reading the instruction, so it degenerates into a vision-only policy that breaks whenever the scene is ambiguous or shifts out of distribution. LangForce aims to recover genuine instruction-following from such biased data by contrasting a vision-only prior against a language-conditioned posterior during training.

Key Contributions

1. Diagnosing the "vision shortcut" / Information Collapse. The authors identify and empirically validate that standard VLA training on goal-driven data drives the conditional mutual information between instructions and actions toward zero, so the policy collapses to the vision-only prior p(a|v).

2. The LangForce framework. They introduce Latent Action Queries — K = 64 learnable tokens appended to a Qwen3-VL backbone — plus a dual-branch Bayesian objective that maximizes the Log-Likelihood Ratio (LLR) between a language-conditioned posterior and a vision-only prior.

3. State-of-the-art simulation results. LangForce reaches a 66.5% average success rate on SimplerEnv and 52.6% on RoboCasa, and the abstract reports an 11.3% improvement on the challenging OOD SimplerEnv benchmark (the contribution list later in the paper states an 8.8% improvement in OOD generalization on SimplerEnv; both figures appear in the text as written).

4. Preservation of general VLM capabilities. Unlike the QwenGR00T baseline, which the authors show degrades into repetitive gibberish on pure-text math prompts, LangForce reportedly retains the underlying VLM's reasoning and language generation ability, attributed to the LLR objective's enforced dependency on language.

Main Findings

• The vision shortcut is measurable in-distribution. On 24 RoboCasa tasks, a model trained with language masked (vision-only) reached 44.6% success versus 47.8% for the language-conditioned baseline — a small gap showing language was largely unnecessary.

• Ambiguity exposes the collapse. On LIBERO, the vision-only model matched the baseline on Spatial (90.2% vs 97.8%), Object (99.6% vs 98.8%), and Long (86.0% vs 92.0%), but collapsed on LIBERO Goal (9.8% vs 97.4%), where multiple tasks share one visual scene.

• Vision shortcuts also exist in diverse, in-the-wild data. Training on BridgeDataV2 and Fractal, the vision-only model reached an action loss of 0.13 versus 0.08 for the full model, yet achieved near 0% success on SimplerEnv.

• Theory: Because the CMI is bounded by H(ℓ|v), and goal-driven data makes H(ℓ|v) ≈ 0, I(ℓ;a|v) is forced to vanish.

• SimplerEnv (WidowX robot, Avg@480): LangForce + Qwen3-VL-4B achieved 66.5% average, ahead of π0.5 (57.1%), Isaac-GR00T-N1.6-Bridge (57.1%), QwenGR00T (55.2%), VideoVLA (53.1%), π0 (53.1%), CogACT (51.3%), Magma (44.8%), RoboVLM (42.7%), OpenVLA-OFT (41.8%), SpatialVLA (34.4%), and TraceVLA (27.7%). Per-task gains include +13.6% on "Put Carrot on Plate" and +15.0% on "Put Eggplant in Yellow Basket."

• RoboCasa (GR1 robot, Avg@50): LangForce reached 52.6%, versus QwenOFT 48.8%, Isaac-GR00T N1.5 48.2%, QwenGR00T 47.8%, Isaac-GR00T N1.6 47.6%, VisionOnly 44.7%, QwenPI 43.9%, and QwenFAST 39.0%.

• Ablation: The dual-branch Bayesian objective is the main driver. The paper states full LangForce at 63.5% versus "+ Action Query" at 57.5% (+6.0%), and the ablation table lists full LangForce at 66.5% with the query-only variant at 57.5% and the QwenGR00T baseline at 55.2%.

• Latent queries are a useful inductive bias on their own. Adding just the Action Query raised the baseline from 55.2% to 57.5% and reduced DiT condition-processing complexity from O(N²) over vision-language tokens to O(K²) over the constant query set.

• Real-world blocks (Franka Research 3): LangForce scored 25/30, 18/30, and 14/30 on in-domain 1-, 2-, and 3-block scenes versus QwenGR00T's 21/30, 14/30, and 9/30; on the unseen Red block OOD setting it scored 9/30 and 3/30 versus 2/30 and 0/30.

• Real-world vegetables: LangForce achieved 97/120 (80.8%) overall versus QwenGR00T's 71/120 (59.2%), with per-object results of eggplant 24/30, pepper 25/30, carrot 27/30, and cucumber 21/30.

Methodology in Plain English

The authors start from a Bayesian decomposition of the policy: π(a|v,ℓ) = p(ℓ|a,v)·p(a|v) / p(ℓ|v). If p(ℓ|v) is sharp, the likelihood term p(ℓ|a,v) adds nothing and the policy just becomes p(a|v). Their fix is to explicitly maximize the log-likelihood ratio log π(a|v,ℓ) − log p(a|v), which is the conditional pointwise mutual information — a bonus paid only when the action explains something about the instruction that the image alone could not.

To compute both distributions with one model, they add 64 learnable "Latent Action Queries" to the VLM vocabulary. These tokens sit as a bottleneck between the VLM and a Diffusion Transformer action head, so only the 64 query hidden states (not the whole token sequence) are fed to the action expert. Causal attention does the rest: in the Priori Branch the input order is [v, Q, ℓ], so the queries cannot see the instruction and encode a pure vision-only prior; in the Posteriori Branch the order is [v, ℓ, Q], so the queries see everything. Both branches share VLM weights. The prior branch's query states are detached from the computation graph when optimizing its flow-matching loss, so the shared backbone is not dragged toward visual shortcuts.

Training combines three terms: a flow-matching action loss on the posterior branch, one on the prior branch scaled by λ = 0.3, and the LLR term scaled by β = 0.1 with a stop-gradient on the p(ℓ|v) baseline so the model cannot "win" by degrading its own language modeling. At inference only the Posteriori Branch runs, so there is no added test-time cost. Training used QwenGR00T from the StarVLA framework, AdamW at learning rate 1e-5 with cosine annealing, DeepSpeed ZeRO-2, gradient clipping at norm 1.0, no gradient accumulation, 8 NVIDIA H100 GPUs, and 50k fine-tuning steps on BridgeDataV2 and Fractal at batch size 16 per device.

Why This Matters

This paper argues that a widely used training recipe — imitation learning on goal-driven robot datasets — structurally prevents language grounding, and that the problem can be mitigated by changing the objective rather than collecting new data. That is valuable because re-collecting large VLA datasets is expensive, and because shortcut behavior can reappear even in broader data whenever optimization exploits local scene-to-task correlations. It also offers an efficiency argument: the query bottleneck decouples action-head cost from the length of the VLM context.

Real-world applications:

• Household and kitchen robots that must interpret "put the bowl in the drawer" versus "put the bowl on the stove" from the same view.
• Warehouse and logistics picking, where the correct target among visually similar objects must be selected from a spoken or written instruction.
• Assistive robotics, where a user's spoken instruction must override whatever the scene makes statistically likely.
• General-purpose robot foundation models that need to retain conversational and reasoning ability after being fine-tuned for control.

Industry relevance: the work targets the post-training stage of VLA development, where teams fine-tune a pretrained VLM on their own narrow task data. LangForce claims no inference-time overhead, meaning it can be slotted into an existing deployment stack without slowing the policy. The finding that standard VLA fine-tuning causes catastrophic forgetting of general VLM abilities is also directly relevant to anyone shipping a robot that is expected to both talk and act.

Future Directions

• Data-centric fixes. The authors explicitly advocate prioritizing ambiguous scenarios during data collection, calling this the long-term, fundamental solution, and note that recollecting or re-annotating existing goal-deterministic datasets at scale is expensive.
• More demanding real-world evaluation. The real-world experiments are limited to pick-and-place and do not validate contact-rich or dexterous manipulation, which the authors leave for future work.
• Training-overhead reduction. The dual-branch design increases per-iteration cost; the paper mitigates it with a prefix prefill strategy that reuses the shared visual prefix, but the exact training-time overhead is not reported.
• Open questions raised by the work: how the approach compares against two-stage Bayesian alternatives such as BayesVLA, which requires freezing the vision prior in a second stage, and how sensitive results are to the number of latent action queries (analysis deferred to Appendix B.6) and to λ and β (Appendix B.5).

Target Audience

Robotics and embodied-AI researchers working on VLA models and imitation learning; machine learning engineers responsible for post-training or fine-tuning robot foundation models on proprietary demonstration data; and readers interested in probabilistic/Bayesian framings of policy learning, information-theoretic objectives, and mitigating shortcut learning and catastrophic forgetting. Readers without background in Bayesian policy factorization, mutual information, or diffusion/flow-matching action heads will find the method sections demanding.

Authors’ abstract

Vision-Language-Action (VLA) models have shown promise in robot manipulation but often struggle to generalize to new instructions or complex multi-task scenarios. We identify a critical pathology in current training paradigms where goal-driven data collection creates a dataset bias. In such datasets, language instructions are highly predictable from visual observations alone, causing the conditional mutual information between instructions and actions to vanish, a phenomenon we term Information Collapse. Consequently, models degenerate into vision-only policies that ignore language constraints and fail in out-of-distribution (OOD) settings. To address this, we propose LangForce, a novel framework that enforces instruction following via Bayesian decomposition. By introducing learnable Latent Action Queries, we construct a dual-branch architecture to estimate both a vision-only prior $p(a \mid v)$ and a language-conditioned posterior $π(a \mid v, \ell)$. We then optimize the policy to maximize the conditional Pointwise Mutual Information (PMI) between actions and instructions. This objective effectively penalizes the vision shortcut and rewards actions that explicitly explain the language command. Without requiring new data, LangForce significantly improves generalization. Extensive experiments across on SimplerEnv and RoboCasa demonstrate substantial gains, including an 11.3% improvement on the challenging OOD SimplerEnv benchmark, validating the ability of our approach to robustly ground language in action.