10 Open Challenges Steering the Future of Vision-Language-Action Models

AI summary

Overview

Research area: Robotics and Embodied AI, specifically Vision-Language-Action (VLA) models that map visual observations and natural language instructions to robot actions.

Technical level: Advanced. The paper is a research roadmap written for the VLA community, containing formal policy objectives, an algorithm sketch with pseudocode, and detailed discussion of competing action-modeling paradigms. It explains concepts as it goes, but assumes familiarity with transformer language models, imitation learning, diffusion models, and robot control terminology.

Scope (one sentence): A position paper that enumerates ten principal challenges facing VLA models, surveys emerging technical trends aimed at those challenges, and proposes an exploratory hierarchical multi-agent framework for organizing VLA research and deployment.

Publication details: arXiv:2511.05936v1 [cs.RO], 08 Nov 2025, licensed CC BY-SA 4.0. Authors are affiliated with Nanyang Technological University, SUTD, Lambda Labs, National University of Singapore, A*STAR, and Stanford University.

What This Paper Is About

VLA models have grown out of the success of LLMs and VLMs, and they let robots follow natural language instructions while generalizing across environments and tasks. The paper argues that despite rapid progress, ten major obstacles stand between today's VLA models and broad real-world adoption, spanning perception, reasoning, data, evaluation, cross-robot generalization, efficiency, whole-body control, safety, multi-agent operation, and human interaction. Rather than reporting new experiments, the authors survey these obstacles, describe four emerging trends that could address them, and sketch a hierarchical planning framework that ties the pieces together.

Key Contributions

1. A structured agenda of ten open challenges. The paper organizes the field's difficulties into ten milestones: multimodality, reasoning, data, evaluation, cross-robot action generalization, efficiency, whole-body coordination, safety, agents, and coordination with humans.

2. A taxonomy of action-modeling strategies. It lays out the split between discrete-action models (e.g., OpenVLA, FAST+ tokenization) and continuous-action models (diffusion- and flow-matching-based action experts), comparing their trade-offs in quantization error, inference speed, and compute budget.

3. Four emerging trends positioned as solutions. Spatial understanding, world modeling, post-training with reinforcement learning, and data synthesis with visual generative models are presented as the pivotal directions for reaching the ten milestones.

4. An exploratory hierarchical multi-agent framework. Algorithm 1 formalizes a workflow with a high-level planner/orchestrator, expert low-level VLA policies, safety guardrails, coordination across subtasks, and goal verification, illustrated with a cooking example and a delivery-driver analogy.

Main Findings

• Depth is largely ignored by VLA models. With the exception of MolmoAct and SpatialVLA, current VLA models do not use explicit depth information. Even in those two cases, depth is used only in the training phase, where the model learns to impute depth from RGB frames, so limitations tied to object size, camera distance, and movement fineness remain.

• Evaluation environments are too controlled. Most VLA models are evaluated on a few real benchmarks involving WidowX or Franka robots in predefined settings. SimplerEval applies distribution shift across background, lighting, distractors, texture, and camera pose, but reflections, lens flares, and environment-induced noise from water, dust, or debris remain unaccounted for.

• Discrete and continuous action models trade off differently. Discretizing actions into 256 bins via a quantile-based strategy introduces quantization errors and slow autoregressive inference, reported at 3-5 Hz, making discrete models unsuitable for high-frequency control. Diffusion-based continuous models predict action chunks jointly and preserve movement fidelity but require a significantly higher compute budget for convergence. Hybrid pipelines that pretrain autoregressively on discrete tokens and then add a continuous action expert show superior performance and significantly faster convergence.

• VLM reasoning does not transfer cleanly to VLA tasks. Models such as Emma-X, CoT-VLA, and MolmoAct, trained on both linguistic and action-level reasoning traces, still produce imperfect results on benchmarks including LIBERO and SimplerEnv, even on simple tasks like picking, placing, operating a drawer, or pointing. Performance generally degrades as task horizon increases.

• High-quality training data remains scarce and noisy. Open-X-Embodiment unifies around 70 and growing smaller datasets covering more than 1M episodes, yet VLA models trained on it remain brittle to out-of-distribution environments and robot setups. Data variance also stems from differences in embodiments, camera positions and angles, and inconsistencies among human data collectors.

• Cross-robot action heterogeneity blocks generalization. Training on action data from a fixed set of embodiments often fails to transfer to robots with different action spaces, such as those with higher degrees of freedom or different structures across arms, quadrupeds, and autonomous vehicles.

• Compute constraints force a capability trade-off. Robots are far more compute-limited than training infrastructure. Server offloading is limited by network latency and disruption, while smaller on-board models are generally less performant than their larger counterparts.

• World models and video generation may fill the data and reward gap. Latent action variables learned from video, V-JEPA-2-style embedding prediction, and generative models that predict next states conditioned on actions offer ways to obtain training data and reward signals without fully engineered simulators.

• Early agentic and safety work exists but is nascent. A heterogeneous framework using an LLM as high-level planner and a VLM as verifier has been proposed to assist a VLA model. Separately, an RL-based approach has been proposed for safety alignment of VLA models by constraining actions while maintaining general performance. Touch-capable VLAs have been shown to enhance overall performance on pick, place, and wipe tasks over vision-only models.

• Reasoning before acting improves performance. CoT-VLA, trained to produce a visual output of the intended state before action decoding, improved overall model performance across standard benchmarks. Emma-X, trained to generate high-level rationale in natural language, outperformed direct action generation.

Methodology in Plain English

This is not an empirical paper; there is no new dataset, no training run, and no benchmark table reported. The authors' method is conceptual synthesis. They first establish background on how VLA models work, describing the standard imitation-learning setup where a policy maximizes the log probability of an action sequence conditioned on observation, language instruction, and prior context, given a demonstration dataset. They then split the field into two action-modeling schools, discrete and continuous, and walk through representative techniques in each. From there, they enumerate ten challenges, citing specific prior systems and benchmarks as evidence for each.

The second half of the paper flips from problems to remedies. The authors propose four trends they consider pivotal, and then assemble an exploratory framework. That framework, described in Algorithm 1, treats each task as a hierarchical planning problem: a high-level planner decomposes a natural language goal into a sequence of subtasks, each subtask is assigned to an expert VLA policy, safety checks are run through a world model simulation, the resulting state feeds the next subtask, and a final check verifies whether the goal has been achieved. A "reasoning before actions" variant inserts an explicit reasoning trace before low-level action generation, illustrated with a cooking example where the system generates guidance such as moving up 22 steps, left 23 steps, down 23 steps, and closing the gripper.

Why This Matters

Impact on research. The paper provides a shared vocabulary for a fast-moving field. By naming ten challenges and mapping concrete prior systems to each, it gives researchers a way to locate their work within a larger agenda rather than optimizing a single benchmark. It also makes explicit a tension the community faces: discrete tokenization is easy to bolt onto language models but slow and lossy, while continuous diffusion preserves motion fidelity but costs far more compute. Flagging the hybrid pretraining-then-continuous-expert recipe as the current best-performing pattern gives a concrete baseline for future comparisons.

Real-world applications cited or implied in the paper:

• Disaster-zone rescue. Embodied systems with audio and speech modalities could listen for victims calling for help or respond to explosive sounds and falling debris, while safety guardrails reduce the risk of a robot harming the people it is trying to save.
• Delicate manipulation. Touch modality would allow VLAs to apply force carefully through joints and gripper, covering handling of glass and ceramic items, cooking, and assembly and disassembly of electronic and electrical devices.
• Mobile manipulation. Whole-body coordination lets a mobile manipulator reposition its base while moving its arm to reach, grasp, or carry objects safely.
• Offline or disrupted environments. On-board, resource-efficient VLA models matter in settings cut off from internet and telecommunication services, where server offloading is not available.

Industry relevance. The compute asymmetry between training clusters and robots directly shapes product architecture. The paper frames embodiments as thin clients that relay observations to servers, which is limited by network latency and disruption, versus on-board models that are smaller and less performant. This trade-off is central to anyone building deployable robots. The safety discussion, motivated by concerns about harmful LLM outputs translating into direct physical harm, points to a regulatory and trust dimension. The data discussion matters commercially because collecting robot teleoperation data is costly, making simulation, video generation, and latent-action extraction attractive for reducing data-acquisition expense.

Future Directions

1. Extending VLA beyond vision and language. The paper calls for incorporating audio, speech, and touch, and asks how evaluation frameworks can absorb these modalities rather than remaining vision-centric.

2. Building depth-aware VLM backbones. A proposed path is fine-tuning VLM backbones on real and synthesized RGB-D data, using frameworks like Locate 3D to synthesize RGB-D frames from RGB frames and inserting new objects to construct depth-requiring QA pairs. Depth estimation experts would remain a fallback for depth-devoid RGB input.

3. Learning universal action representations for zero-shot robot transfer. The paper notes that codebook-based universal atomic actions reduce adaptation effort and data requirements but still fall short of true zero-shot generalization. It envisions teaching a VLA a new robot's action space through prompting, analogous to in-context learning in LLMs, which would require a fundamental shift in VLA pretraining.

4. Scalable post-training without engineered simulators. Defining and providing reward signals for VLA models is described as a fundamental open challenge. World models and video generative models are proposed as implicit reward estimators, with preference optimization methods such as DPO, RL, and GRPO suggested for improving the action expert. Evaluators would also serve as virtual guardrails, which the paper notes is especially important in multi-agent settings where interactions can introduce emergent risks.

Target Audience

This paper is most useful to robotics and embodied AI researchers, particularly those working directly on VLA models, action tokenization, or robot policy learning, who want a consolidated map of where the field's unsolved problems lie. It also suits graduate students entering the area and needing orientation, engineering teams deciding between discrete and continuous action architectures or between on-board and server-offloaded inference, and research program managers or funders looking for a structured agenda across perception, data, evaluation, safety, and multi-agent coordination.

Authors’ abstract

Due to their ability of follow natural language instructions, vision-language-action (VLA) models are increasingly prevalent in the embodied AI arena, following the widespread success of their precursors -- LLMs and VLMs. In this paper, we discuss 10 principal milestones in the ongoing development of VLA models -- multimodality, reasoning, data, evaluation, cross-robot action generalization, efficiency, whole-body coordination, safety, agents, and coordination with humans. Furthermore, we discuss the emerging trends of using spatial understanding, modeling world dynamics, post training, and data synthesis -- all aiming to reach these milestones. Through these discussions, we hope to bring attention to the research avenues that may accelerate the development of VLA models into wider acceptability.