Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models

AI summary

Overview

• Research area: Robotics, specifically Vision-Language-Action (VLA) models — how to continued-pre-train a vision-language model (VLM) backbone so it becomes a better foundation for robot policies.
• Technical level: Advanced. The paper assumes familiarity with VLM/VLA architectures, action heads (discrete token, regression, flow-matching, diffusion-style continuous), and robot benchmarks; the pilot study and design principles are explained clearly enough for a non-specialist to follow the argument.
• Scope: The paper introduces VLAct, a representation-centric continued-pre-training recipe for VLA backbones, and evaluates it on multi-embodiment simulation benchmarks, real-world Franka experiments, and unseen-embodiment transfer using fully open-source data and a 16-GPU setup.

What This Paper Is About

Robot trajectories are far harder to scale than web image-text data because they must be collected through physical embodiment, so even large robot datasets remain sparse samples of the physical interaction space. The paper argues that under a fixed robot-data budget, the bottleneck is not only how many trajectories are collected but how effectively continued pre-training converts them into transferable visual-action representations. The goal is to design a continued-pre-training recipe that produces a reusable VLM backbone — one that preserves the broad VLM prior, avoids over-specializing to a single action head, and shares action semantics across embodiments — rather than a policy initialization tied to one head.

Key Contributions

1. A pilot study isolating the action-head axis. Holding the VLM backbone fixed to Qwen3-VL-4B, the authors vary the action head used during pre-training and fine-tuning on LIBERO-Plus and a RoboTwin-Clean setting, and identify two failure modes: discrete FAST supervision transfers across heads but loses fine-grained temporal and amplitude information, while single-head continuous (OFT) supervision induces head-specific representation collapse — strong same-head performance but poor transfer to PI or GR00T heads.

2. VLAct, a representation-centric continued-pre-training recipe with three components: VLM-prior preservation through shallow-layer protection and caption-data mixing; multi-head continuous action co-supervision using OFT, PI, and GR00T heads on a shared backbone latent; and a partially unified cross-embodiment action layout with a wrap-aware loss for periodic joint angles.

3. A design that separates pre-training from downstream use. The pre-training heads and caption stream are discarded after continued pre-training; downstream users attach a freshly initialized task-specific action head and fine-tune under their own protocol. In every VLAct comparison, the VLM backbone weights are the only thing that changes.

4. An open, modest-compute demonstration. All models and training pipelines are open-sourced, pre-training uses fully open-source data (DROID, InternA1, RoboCoin, MolmoAct plus captioning data) and a 16-GPU setup, built on the StarVLA training codebase with Qwen3-VL-4B as the base backbone.

Main Findings

• LIBERO-Plus: 82.6% overall success, the best total score in the comparison. This improves over the strongest in-house baseline, Qwen3VL-OFT, by 7.6 points (82.6% vs. 75.0%) under an identical backbone family and OFT downstream head, and surpasses Abot-M0 (80.5%) by 2.1%. The largest gains over Qwen3VL-OFT appear on the Camera, Robot, Noise, and Layout perturbation dimensions.

• VLA-Arena: 54.8% success, as reported in the paper's highlights.

• RoboTwin 2.0: strongest in the Base setting and competitive in Data Scaling. VLAct-OFT reaches 80.5% Clean and 41.5% Random with only 50 clean trajectories per task, and 92.5% Clean and 90.8% Random when 500 randomized trajectories per task are added. The PI head attains the highest Clean success in the Data Scaling setting at 93.0%, slightly above the default OFT head at 92.5%, and all three VLAct heads remain within 3.4 points of each other — supporting the head-transfer claim.

• Real-world Franka experiments: consistent gains over a non-pre-trained Qwen3VL-4B-OFT baseline. On single-arm short-horizon in-domain tasks VLAct reaches 92.5% average success versus 77.5%. On out-of-domain novel-object tasks it reaches 90.0% on both "novel object from pot" and "novel object in cup" versus 73.3% and 65.0%. On long-horizon tasks it scores 86.6% on table cleaning and 80.0% on scooping beans versus 73.3% and 33.3%, and under harder OOD settings 82.5% and 83.3% versus 47.5% and 46.6%. Despite pre-training only on single-arm data, it reaches 72.0% average on dual-arm coordination tasks versus 44.0%.

• RoboCasa-GR1: transfer to a humanoid embodiment never seen during continued pre-training. With full fine-tuning data VLAct reaches 54.0%, above Qwen3VL-OFT at 48.8%, GR00T-N1.6 at 47.6%, and π0.5 at 37.0%. With only 20% of the trajectories it already reaches 49.5%, exceeding the full-data GR00T-N1.6 baseline at 47.6%; with 50% of the data it reaches 51.0%.

• RoboDojo: 10.66 average score and 7.60% success rate, ranking eighth by average score and sixth by average success rate among 35 policies on the official August 24, 2026 leaderboard snapshot. Among the four explicitly designated world-action-model (WAM) entries, X-WAM is the strongest at 7.69 score and 3.83% success; VLAct exceeds X-WAM by 2.97 score points and 3.77 percentage points in success, outperforming every designated WAM entry on both aggregate metrics, and also outperforms industry systems including Xiaomi-Robotics-0, GalaxeaVLA (G0), LingBot-VLA, and ABot-M0. Largest gains are on Precision and Long-Horizon tasks; memory remains a clear limitation.

• Gains are attributable to the backbone. Across every VLAct results comparison, the action head and its fresh initialization, the downstream data, the optimizer, and the fine-tuning budget are identical, so the reported 7.6–21.4 point gains come from the backbone alone.

• Ablation: shallow-layer protection helps. Freezing the vision encoder and the lower half of the LLM layers during VLA pre-training improves downstream performance by 3.7% on LIBERO-Plus and 3.4% on Agilex.

• RoboDojo compared to a same-backbone entry. Against the Qwen3-VL-based StarVLA-α entry, VLAct improves average score by 4.26 points and success by 4.36 percentage points.

Methodology in Plain English

The authors start from an already pretrained VLM (Qwen3-VL-4B) rather than training a foundation model from scratch, and continued-pre-train it on broad, heterogeneous, multi-embodiment robot trajectories before any task-specific fine-tuning — the same setting that systems such as π0, π0.5, and GR00T N1/N1.5 use.

They first run controlled pilot experiments to isolate how the choice of action head shapes the backbone. Fixing the backbone, they swap discrete FAST heads and continuous heads (OFT, PI, GR00T) and observe that discrete supervision teaches coarse action structure but loses fine-grained detail, and that single-head continuous supervision produces features organized around that one head's geometry.

The recipe then has three parts. First, to keep the VLM prior from being overwritten by narrow robot data, they freeze the entire vision encoder and the lower half of the LLM layers while updating only the upper LLM layers and action heads, and they mix captioning data into pre-training (they ablate several VLM data mixtures and find captions the strongest anchor). Second, to prevent collapse toward one decoder, they attach three continuous heads — OFT, PI, and GR00T — to the same shared backbone latent and sum their losses on the same ground-truth action chunk, so the backbone must expose action information usable by several parameterizations; because all heads share one backbone forward pass, this adds only lightweight head-specific computation. Third, to share action supervision across embodiments, they use one shared head with a partially unified action space: gripper dimensions are shared across robots because open/close semantics are comparable, while arm dimensions stay embodiment-specific, and each sample contributes loss only on active dimensions with inactive ones masked out. No embodiment adapter, routing module, or embodiment-conditioned decoder is introduced. They also apply a wrap-aware loss on absolute joint-angle dimensions, measuring angular residuals modulo 360°, because standard regression treats 179° and −179° as 358° apart although they are physically 2° apart.

During fine-tuning, all of this is discarded: the pre-training heads and caption stream are removed, a freshly initialized task-specific head is attached, and training proceeds under the same downstream setting as each baseline. The vision encoder and lower layers that were frozen are unfrozen so the full model can adapt. Real-world experiments use Franka Research 3 7-DoF arms, with one model trained for all single-arm tasks and another for all dual-arm tasks, each for 50k training steps on 8 H800 GPUs and evaluated over 10 trials.

Why This Matters

Impact on research. The paper reframes the VLM backbone as a first-order design variable for VLA rather than a fixed component inherited from general vision-language pre-training. It offers evidence that representation-centric continued-pre-training design is an independent axis of progress alongside data scaling, and it does so with open data and a 16-GPU budget, in contrast to frontier systems such as the π series that often rely on closed-source data. It also gives practitioners a concrete diagnostic (head-transfer and same-head performance can diverge) and a simple intervention (use head diversity itself as supervision).

Real-world applications.

• General-purpose manipulation arms: single-arm Franka tasks such as cube stacking, pen-in-cup, table cleaning, and scooping beans, with gains reported especially on spatially sensitive and long-horizon tasks.
• Dual-arm and bimanual robots: AgileX-style bimanual manipulation and coordination-heavy tasks such as deformable-object folding (pants, towel) and breakfast preparation.
• Humanoid robots: transfer to the held-out GR-1 humanoid under a limited downstream data budget.
• Robustness-oriented deployment: LIBERO-Plus and RoboTwin 2.0 Random measure behavior under camera, lighting, background, noise, layout, and instruction perturbations, which matters for real environments that do not match training conditions.

Industry relevance. VLAct is compared directly against industry-developed and large-scale systems — Abot-M0, LingBot-VLA, InternVLA-A1, Being-H0.7, Motus, Fast-WAM, HoloBrain-0-QW, Xiaomi-Robotics-0, GalaxeaVLA, DM0.5, and others — and reports competitive or better results on several benchmarks. The paper notes that on RoboDojo six of the seven higher-scoring entries come from industry teams or robotics companies (Dexmal, Galaxea AI, Xiaomi Robotics, Tencent Robotics X, OpenHelix Robotics, and Physical Intelligence), and that the leaderboard does not normalize for training compute. The takeaway for industry is that continued-pre-training design, not just trajectory volume or compute, can substantially change what a fixed resource budget delivers.

Future Directions

• Memory and longer-horizon reasoning. The authors state that Memory remains a clear limitation on RoboDojo, and the largest gains there come from Precision and Long-Horizon tasks — leaving memory-centric and long-horizon reasoning as open problems.
• Normalized and controlled comparison against industrial systems. The RoboDojo leaderboard does not normalize for training compute, and the paper notes that training compute, optimization schedules, and checkpoint-selection procedures may differ across methods in the RoboTwin 2.0 Data Scaling comparison, so compute-matched evaluation is an open question.
• How far representation-centric design extends beyond data scaling. The paper presents its recipe as an independent axis alongside scale; the natural next step is determining how the recipe interacts with larger robot-data budgets, additional embodiments, and additional action-head families beyond OFT, PI, and GR00T.
• Making foundation VLMs better physical-task foundations. The authors explicitly hope to encourage the community to study how foundation VLMs can be made better suited to physical tasks, rather than treating the VLM as an inherited fixed component — which implies further work on prior preservation and action-aware representation design.

Target Audience

Researchers and engineers working on VLA models, robot policy pre-training, and cross-embodiment transfer who need to decide how to spend a fixed robot-data and compute budget. It is also useful for practitioners who must attach their own action head to a pretrained backbone, and for readers tracking the debate over whether VLA progress is primarily a data-scaling problem. The pilot study and the plain-language design rationale make the core argument accessible to graduate students entering the area, while the benchmark tables and implementation details (shallow-layer protection, multi-head co-supervision, partially unified action space, wrap-aware loss) are aimed at specialists.

Authors’ abstract

Scaling robot data is crucial for building generalist Vision-Language-Action (VLA) models, yet robot trajectories are harder to scale than web-scale image-text data because embodied collection is costly and sparsely covers the physical world. This makes representation quality a central bottleneck: under a fixed robot-data budget, continued pre-training must turn limited trajectories into transferable visual-action knowledge rather than merely fit actions. We propose VLAct, a VLA-oriented VLM backbone trained on broad, heterogeneous, multi-embodiment robot data before task-specific fine-tuning. VLAct preserves the broad VLM prior and encourages shared action semantics across embodiments through VLM-prior preservation, multi-head continuous action co-supervision, and a partially unified cross-embodiment action layout, while allowing task-specific action heads during fine-tuning. Across simulation, real-world, and unseen-embodiment transfer, VLAct consistently improves downstream performance under fixed fine-tuning protocols. On LIBERO-Plus and RoboTwin 2.0, VLAct surpasses industrial VLA systems including ABot-M0 and LingBot-VLA, achieving success rates of 82.6% and 92.5%. On RoboDojo, VLAct ranks sixth among all policies by success rate and outperforms all explicitly designated world-action model (WAM) entries on both metrics. Most notably, on RoboCasa-GR1, an unseen humanoid embodiment, VLAct using only 20% of downstream trajectories outperforms the full-data GR00T-N1.6 baseline. These results are obtained using fully open-source data and only a 16-GPU training setup, showing that representation-centric continued pre-training can deliver highly competitive performance under a modest compute budget and is an important independent axis of VLA progress beyond data scaling.