VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models

AI summary

Overview

Research area: Robotics — benchmarking and evaluation of Vision-Language-Action models (VLAs), the class of models that map camera images and natural-language instructions to robot control actions.

Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with robot policy learning, simulation benchmarks, and terms such as fine-tuning, distribution shift, and tokenization of actions.

One-sentence scope: VLA-Arena is an open-source benchmark, toolchain, and dataset suite that systematically varies task structure, language commands, and visual observations to measure not only whether current VLA models succeed, but how and where they fail.

What This Paper Is About

Vision-Language-Action models are advancing toward general-purpose robot policies, but the field lacks a rigorous way to characterize their capability boundaries and failure modes. Existing simulation benchmarks tend to use static tasks at a single fixed complexity, usually separate robustness-to-noise from task extrapolation, and largely ignore safety constraints. The authors build VLA-Arena to quantify difficulty along three orthogonal axes — task structure, language command, and visual observation — so that model capability frontiers can be measured precisely rather than summarized by a single success number.

Key Contributions

1. A structured benchmark with graded difficulty. VLA-Arena comprises 11 task suites organized into four dimensions — Safety, Distractor, Extrapolation, and Long Horizon — totaling 170 tasks. Each suite spans three difficulty levels (L0–L2), and fine-tuning is restricted to L0 so that L1 and L2 measure genuine generalization. Orthogonal, task-independent perturbation axes for language (W0–W4) and vision (V0–V4) act as diagnostic probes.

2. A formal task and safety specification language. The benchmark is defined in a constrained behavior domain definition language (CBDDL), which extends BDDL with native support for dynamic entities, perturbations, and formal safety constraints. CBDDL defines 10 distinct types of safety predicates, split into instantaneous checks (e.g., collision, force limits) and terminal checks (e.g., object dropping), and enables a cumulative cost (CC) metric that sums instantaneous violations over a trajectory plus a terminal term scaled by a factor of 10.

3. An extensive empirical study of leading VLAs. The paper evaluates six models spanning the two dominant architectural paradigms — autoregressive action generation (OpenVLA, UniVLA, π0-FAST) and continuous action generation (π0, OpenVLA-OFT, SmolVLA) — plus an appendix table covering six newer models (π0.5, GR00T-N1.6, Qwen2.5-VL-OFT, Qwen3-VL-OFT, Qwen2.5-VL-GR00T, Qwen3-VL-GR00T).

4. An end-to-end open-source release. The authors provide a complete toolchain from scene definition through data collection, processing, training, and automated evaluation, along with human-collected VLA-Arena-S/M/L fine-tuning datasets (10, 30, and 50 trajectories per task respectively), tutorials, documentation, and a public leaderboard at https://vla-arena.github.io.

Main Findings

• Memorization beats generalization. Models fine-tuned on L0 tasks perform well in distribution but degrade sharply on near-distribution (L1) and far-distribution (L2) challenges across all four dimensions. For example, in the Safety suite StaticObstacles, π0 scores 0.97 / 0.66 / 0.33 success rate across L0–L2, and OpenVLA scores 0.6 / 0.6 / 0.0.

• Skills do not transfer through language. Despite mastering atomic skills at L0, models fail when language requires adapting skills to novel contexts (Extrapolation) or novel sequences (Long Horizon). Performance drops to near-zero once instructions require composing or reordering concepts; models succeed mainly when the instruction's semantic structure stays fixed, such as UnseenObjects-L1.

• Fragile visual selective attention. Under the Distractor dimension, performance falls sharply with static distractors as models fail to separate targets from clutter and grasp the wrong object. Unexpectedly, models are less sensitive to dynamic distractors, which the authors attribute to the visual distinctiveness of moving objects and their lack of stable grasp affordances.

• Visual grounding collapses under viewpoint and noise shifts, unlike a VLM. Qwen3-VL-8B keeps 100.0% grounding accuracy at V0–V2, 96.7% at V3 (3.3% drop), and 93.3% at V4 (6.7% drop). The averaged VLA policies show 13.5% drop at V1, 24.0% at V2, 30.5% at V3, and 50.5% at V4. The authors read this gap as catastrophic forgetting: fine-tuning discards generalizable visual concepts in favor of specific pixel distributions.

• Apparent language robustness can be a visual shortcut. On StatePreservation, model performance is flat across language perturbations — but the models also collapse on UnseenObjects, where language is required to disambiguate the target. The authors conclude that the flat curve reflects reliance on visual shortcuts rather than genuine instruction grounding.

• Safety is neglected and traded against success. High success rates are frequently achieved while violating safety constraints, producing high cumulative cost. In Table 2, for instance, StaticObstacles L1 cumulative cost is 40.2 for OpenVLA-OFT and 64.6 for UniVLA, and CautiousGrasp L1 cost is 96.9 for OpenVLA.

• Rank reversals across difficulty levels. The disentangled leaderboard in Table 4 shows that strength at one level does not predict strength at another: OpenVLA ranks first at L2 but is unranked at L0, UniVLA ranks first at L1 but is unranked at L2, and π0 ranks first at L0 and in the combined L0+L1+L2 overall ranking but third at L1 and L2. SmolVLA ranks first on Safety Cost while OpenVLA-OFT ranks third.

• Task language matters more than in LIBERO. When instructions are invalidated, LIBERO drops only 28%, while VLA-Arena collapses 52–64% from a 79% baseline, indicating that VLA-Arena tasks are more deeply language-grounded rather than solvable through visual shortcuts.

• Real-robot results mirror the simulation. On a physical Franka Research 3 with π0 and five other baselines, average success falls from 60.0% at L0 to 26.6% at L1 and 3.3% at L2. Unsafe behaviors appear at L1 (6/10) and L2 (4/10) despite 0/10 at L0. Physical lighting and camera changes reduce average success from 60.0% (canonical) to 37.0% (+Light) to 20.0% (+Light+Camera), tracking the simulated visual perturbation trends.

Methodology in Plain English

The authors build their benchmark on the MuJoCo-based robosuite and LIBERO simulation stack, but replace the usual flat task list with a structured design. Every task is written in CBDDL, an extended task-description language that can also declare safety constraints, so safety can be scored as an objective separate from task success.

Tasks are organized into three difficulty levels. L0 replicates the training distribution with direct instructions and simple scenes. L1 introduces controlled variations — more objects, new instances of known categories, novel concept combinations, moderate clutter, or simple safety constraints. L2 introduces structural changes: different workflows, unusual object arrangements, dense clutter, strict safety constraints, or entirely new object categories.

Two extra axes are layered onto any task. The language axis substitutes key words with WordNet-derived near-synonyms (synsets connected by a shortest path length of 1), from W0 (original instruction) to W4 (four substituted slots). The visual axis applies a cumulative hierarchy: V1 adds randomized brightness, contrast, saturation, and temperature; V2 adds randomized object colors; V3 adds randomized camera positions; V4 adds Gaussian noise to images.

Evaluation reports success rate and cumulative cost, averaged over 30 episodes with 10 episodes per seed. For fine-tuning, the authors supply human demonstrations re-rendered at 256×256 with rotation correction, adaptive filtering of grasping no-ops, and standardized third-person and wrist views, in three sizes (10, 30, 50 trajectories per task). The standard protocol fine-tunes on VLA-Arena-L0-L and evaluates generalization on L1 and L2. Finally, they replicate core suites on a physical Franka Research 3 robot to check whether the simulated failure modes carry over.

Why This Matters

Impact on research. VLA-Arena shifts evaluation from a single success-rate leaderboard to a diagnostic profile that separates memorization from generalization, exposes visual shortcuts, and scores safety independently. The paper's rank reversals across L0–L2 show that a single difficulty level gives an incomplete picture, which argues for structured difficulty design as a standard practice and gives the community a reproducible substrate and curated datasets for fair comparison.

Real-world applications:

• Household and service robots that must manipulate containers without spilling, avoid fragile objects, and follow user instructions phrased in different ways.
• Industrial and warehouse manipulation where collision-free motion around static and moving obstacles is a hard requirement.
• Assistive and care robotics, where handling hazardous items such as sharp implements by safe contact points is directly analogous to the CautiousGrasp suite.
• Deployment-readiness testing: the simulated vision perturbations act as a lower-bound proxy for real sensor degradation, lighting changes, and camera calibration error.

Industry relevance. The paper's central claim — that high task success often comes alongside safety violations — speaks directly to the deployment gap for commercial robotics. A benchmark that produces cumulative cost alongside success rate gives engineering teams a way to gate releases on safety rather than on task completion alone, and the released toolchain lowers the cost of setting up such evaluations.

Future Directions

• Mitigating catastrophic forgetting during fine-tuning. The gap between Qwen3-VL-8B's grounding accuracy and VLA policy performance suggests methods are needed to preserve pretrained visual representations while adapting to robot action data.
• Language-driven skill composition. Because models reduce varied instructions to a fixed set of behavioral patterns, a key open problem is training or prompting policies that genuinely compose known skills into novel workflows and spatial relations.
• Safety-aware training objectives. Cumulative cost is currently measured, not optimized; training policies against explicit safety predicates defined in CBDDL is a natural next step.
• Closing the sim-to-real gap for diagnostic perturbations. The authors present simulated perturbations as a lower bound for real fragility; establishing how tightly simulated degradation predicts physical degradation across more robots and environments remains open.

Target Audience

This paper is most useful to robotics and embodied-AI researchers who train or evaluate VLA policies, to benchmark designers looking for a template for structured difficulty and safety scoring, and to industry practitioners assessing whether a policy is ready for physical deployment. Readers focused on robot safety, generalization, and sim-to-real transfer will find the diagnostic framing and the released toolchain especially relevant.

Authors’ abstract

While Vision-Language-Action models (VLAs) are rapidly advancing toward generalist robot policies, quantitatively characterizing their capability boundaries and failure modes remains challenging. To address this, we introduce VLA-Arena, a comprehensive benchmark. It features a novel structured task design framework to quantify difficulty across three orthogonal axes: (1) Task Structure, (2) Language Command, and (3) Visual Observation. This allows us to systematically design tasks with fine-grained difficulty levels, enabling a precise measurement of model capability frontiers. For task structure, VLA-Arena comprises 11 task suites organized into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon, totaling 170 tasks. Each suite spans three difficulty levels (L0-L2), with fine-tuning restricted to L0 to rigorously assess generalization. Orthogonal to this, language (W0-W4) and visual (V0-V4) perturbations can be applied to any task as diagnostic probes to distinguish robust grounding from superficial pattern matching. Our extensive evaluation of state-of-the-art VLAs reveals critical limitations: memorization over generalization, superficial visual perception, and a neglect of safety constraints. Additionally, model rank reversals across L0-L2 validate that each level provides non-redundant insights. To foster research addressing these model limitations and ensure reproducibility, we provide the complete VLA-Arena framework, including an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L datasets for fine-tuning. Our benchmark, datasets, models, and leaderboard are publicly available at https://vla-arena.github.io.