From Pixels to Actions - Towards Native Vision-Language-Action Policies

AI summary

Overview

Research area: Robot learning / Vision-Language-Action (VLA) models, at the intersection of computer vision, multimodal foundation models, and embodied control.

Technical level: Advanced. The paper assumes familiarity with transformer backbones, vision encoders, flow matching, and robot imitation-learning benchmarks.

Scope: An empirical and architectural study of NEO-VLA, an encoder-free 2B-parameter VLA that feeds raw pixels into a shared vision-language backbone and lets a flow-matching action expert read intermediate backbone layers, with controlled comparisons against encoder-based backbones and evaluations on LIBERO, LIBERO-Plus, LIBERO-Pro, RoboTwin 2.0, and a real-world setup.

What This Paper Is About

Most VLA models push images through a separately pretrained vision encoder (CLIP, SigLIP, or DINOv2) and only then hand a fixed feature output to a language backbone and an action head. The authors argue that manipulation depends on exactly the signals this bottleneck tends to lose: local geometry, boundaries, depth, and subtle state changes. They ask whether an action policy should inherit a fixed visual endpoint or instead directly access the visual computation as it evolves across backbone layers, and they build NEO-VLA to test that question.

Key Contributions

1. A structurally native VLA policy. NEO-VLA combines a lightweight convolutional pixel-to-token embedder (overall stride 32, one token per 32×32 image region), a 40-layer, 2B-parameter encoder-free vision-language backbone, and a flow-matching action expert attached to configurable backbone layers — with no separately pretrained vision transformer, image codebook, or autoencoder.

2. A systematic study of action readout depth. Matched-capacity layer-map comparisons (first-28, skip-28, end-28, plus disjoint first-14/last-14) and attachment-point interventions (key/value swaps, input masking) isolate where action-relevant visual information lives.

3. Matched backbone-family comparisons. NEO-VLA is compared against Qwen3-VL-2B and Gemma-3-4B using the same action expert, the same robot-training corpus, the same Open X-Embodiment 75k-iteration pretraining, and the same LIBERO evaluation budget.

4. Broad benchmark and real-world evaluation. Results on LIBERO, LIBERO-Plus (seven perturbation axes), LIBERO-Pro (four perturbation types), RoboTwin 2.0 (clean and Clean2Rand), plus a three-task real-world benchmark with 50 demonstrations per task.

Main Findings

• Native backbone transfers better when frozen. With only the action expert trained, NEO-VLA reaches 90.9% average LIBERO success versus 79.5% for Qwen3-VL-2B and 84.5% for Gemma-3-4B. After full fine-tuning: 95.3% (NEO-VLA), 90.8% (Qwen3-VL), 91.0% (Gemma-3). On LIBERO-Long the frozen NEO-VLA score of 85.4% exceeds the full-fine-tuned encoder-based results of Qwen3-VL (78.6%) and Gemma-3 (82.8%).

• Exposing encoder internals is not equivalent. Routing 13 expert layers to Qwen3-VL's intermediate vision-tower features lowers frozen-backbone LIBERO average from 79.5% to 64.9%, and only matches the original interface under full fine-tuning (90.9% vs 90.8%). The authors conclude the benefit comes from how encoder-free pretraining shapes representations, not merely from earlier feature access.

• Early layers matter most when the backbone is frozen. With a frozen backbone, first-28 scores 94.9% and skip-28 90.9%, versus end-28 at 86.7%. The disjoint comparison agrees: first-14 reaches 93.6% and last-14 89.6% (though first-14 also shortens the backbone).

• Fine-tuned policies prefer distributed readouts. After full fine-tuning, skip-28 is best at 95.3%, ahead of first-28 (94.6%) and end-28 (92.5%).

• Interventions confirm early-layer dominance. Swapping cached keys and values at layers 0–10 causes the largest translation errors, while matched-magnitude residual noise stays within 0.004 cm of the unchanged error — so content, not disturbance size, drives the effect. Wrist-image masking peaks around layers 6–10; instruction masking peaks at layer 0 with a smaller rise near layer 17.

• Multiple capabilities peak at different depths. Visual, linguistic, and geometric signals have distinct depth profiles, so no single readout layer captures all of them.

• Preserving multimodal understanding protects action robustness. Co-training with understanding data keeps MMBench-EN within 1.5 points of baseline (82.7 → 81.2) and yields 69.4% on RoboTwin 2.0 Clean2Rand. Action-only fine-tuning loses valid text generation entirely and collapses to 10.7% on Clean2Rand.

• Strong in-distribution LIBERO performance. NEO-VLA averages 97.4% at step 27,500 (Spatial 98.4, Object 99.4, Goal 95.6, Long 96.2), in the same range as EO-1 (98.4), MMaDA-VLA (98.0), and Qwen-RobotManip (99.1 aggregate), with the authors cautioning that LIBERO is increasingly hard to use for distinguishing top policies.

• Best overall LIBERO-Plus score among open models. NEO-VLA reaches 89.7 total, with 94.5 under camera shift, 91.7 under language shift, 96.9 under image noise, and 96.6 under background change. Gains are smaller under robot (72.6) and layout (82.4) shifts, which change embodiment or scene configuration rather than the observation itself.

• LIBERO-Pro shows a clear split. Semantic robustness is highest in the table at 98.0, and object perturbation is 84.3, but position (3.0) and task (6.3) perturbations collapse, giving an average of 47.9 — below π0.5's 53.3.

• RoboTwin 2.0 shows modest degradation under randomization. Clean 75.78, Clean2Rand 69.40, the strongest Clean2Rand among methods with public code and weights, though its clean score trails closed models such as Qwen-RobotManip (89.58).

• Real-world results lead the compared methods. Across three tasks with 50 demonstrations each and 20 trials per task, NEO-VLA scores 12/20, 17/20, 13/20, versus π0.5 at 10/20, 14/20, 13/20 and StarVLA at 1/20, 2/20, 0/20.

Methodology in Plain English

The team removed the standalone vision encoder. Raw images from up to three camera views are cut into patches and passed through a small two-stage convolutional embedder (stride 32), so each 32×32 image region becomes one token. Images keep their native aspect ratios and are resized to fit a budget of up to 88 visual tokens per view. Camera intrinsics and rotations go through small MLPs, and six extra tokens encode previous and current robot states for Arm 0, Arm 1, and the body. Everything — pixels, camera geometry, state tokens, and a structured text prompt covering control frequency, embodiment, gripper configuration, frame reference, and instruction — enters a single 40-layer, 2B-parameter backbone, where text attends causally and visual tokens attend bidirectionally within each visual block.

Actions are not tokenized as text. A separate, narrower action expert attends to keys and values cached from chosen backbone layers and predicts a continuous velocity field via flow matching, covering both arms (7 joint targets, 3 translation increments, 3 rotation-vector increments, 1 gripper, 12 hand-joint targets each) and the body (3 base-motion, 3 torso, 2 head channels) over 30-step chunks. Training uses τ = 0.999u with u ~ Beta(1, 1.5), group loss weights (2, 1, 1, 1), and 10 Euler steps at inference with a fixed prefix KV cache and no classifier-free guidance.

Because the expert's attachment points are an explicit architectural knob, the researchers define layer maps — first-28 (layers 0–27), skip-28 (28 layers spread over all 40), end-28 (layers 12–39), and full-40 — and compare them with matched expert capacity. They then intervene directly: swapping one attachment's cached keys and values with those from a different observation, and masking individual input spans at single attachment points.

The backbone itself was trained in three stages: large-scale image-text pretraining, mid-training on captioning/conversation/OCR/detection data, then supervised instruction tuning. Robot training adds three more stages: 113,400 steps with the backbone frozen (batch 8192, peak LR 2e-4), 75,000 steps of full fine-tuning (expert LR 2e-4, vision-language LR 5e-5) with understanding data interleaved every other step at auxiliary loss weight 0.1, then 75,000 steps on in-house action data. The pretraining corpus combines nine public dataset families totaling 2,231,583 episodes and 9,927.7 hours, with a filtering pass that masks implausible joint readings (outside ±2π, or more than π from a five-row local median), channels constant across at least 90% of episodes, action outliers, corrupt episodes, and suspect camera calibrations.

Why This Matters

Impact on research. The paper reframes a design assumption that most VLA work leaves implicit. Rather than accepting the encoder output as a fixed interface, it treats visual readout depth as a first-class policy choice and provides evidence that action-relevant signals peak before the final representation. Its control — routing expert layers into Qwen3-VL's own vision tower and finding performance drops — is the sharpest result, because it rules out the simple explanation that "earlier features are just better."

Real-world applications:

• Warehouse and logistics picking, where camera angle, lighting, and background vary constantly and policies must survive perceptual shift.
• Household service robots handling multi-step tasks such as placing objects precisely or reorienting items, which the real-world benchmark directly targets.
• Manufacturing and assembly, where fine-grained geometry and boundaries matter more than broad semantic recognition.
• Cheaper deployment on edge hardware, since the design removes a separately pretrained vision encoder and demonstrates competitive control at 2B parameters against larger models.

Industry relevance. A 2B model that can be trained frozen-first (stage 1 updates only the expert) is attractive for teams that cannot afford full-stack retraining, and the finding that action-only fine-tuning destroys understanding gives a concrete argument for co-training pipelines. The same result is a warning: optimizing purely for task success can silently degrade the backbone's other capabilities.

Future Directions

• Closing the spatial and task-level gap. LIBERO-Pro position (3.0) and task (6.3) scores show the policy still fails when success requires adapting to unfamiliar spatial configurations or new goals, even though semantic robustness reaches 98.0.
• Learning the layer map instead of choosing it. The paper shows fixed maps trade off differently between frozen and fine-tuned regimes; adaptively selecting or weighting attachment depths is a natural next step.
• Explaining why encoder-free pretraining produces control-ready features. The authors demonstrate the effect but cannot fully attribute the LIBERO gains, since pretraining data and capacity differ from Qwen3-VL and Gemma-3.
• Extending beyond the tested embodiment interface. Robot and layout perturbations remain the weakest LIBERO-Plus axes (72.6 and 82.4), suggesting that cross-embodiment generalization is unresolved.

Target Audience

Robotics and embodied-AI researchers building VLA policies; multimodal foundation model researchers interested in encoder-free architectures; engineers deploying manipulation policies under real perceptual variability; and graduate students who already understand transformer backbones and imitation learning and want a clear, controlled case study in how interface design shapes policy capability.

Authors’ abstract

Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. More strikingly, layer-wise probing shows that the action-critical signals often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.