ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learning

AI summary

Overview

Research area: Robotics and embodied AI — specifically Vision-Language-Action (VLA) foundation models for robot manipulation, positioned within the "one-brain, many-forms" paradigm of general-purpose embodied agents.

Technical level: Advanced. The paper assumes familiarity with VLA architectures, diffusion/flow-matching generative models, diffusion transformers (DiT), vision-language model backbones, and robot action representations (end-effector frames, rotation vectors, delta actions).

Scope: The paper describes a complete pipeline — data curation, standardization, model architecture, and training strategy — for building a cross-embodiment VLA foundation model, validated on four manipulation benchmarks.

What This Paper Is About

Building embodied agents that work across many different robot bodies is blocked by fragmented data, inconsistent action/coordinate conventions, and training objectives inherited from vision-language models that emphasize semantic recognition rather than 3D structure and physical dynamics. The authors integrate six public robot datasets into a single standardized corpus and pair it with a model that predicts clean robot actions directly rather than predicting noise or flow velocity, aiming to make action decoding faster and policies more stable. They also study how the choice of sampling strategy across datasets and embodiments affects cross-embodiment and cross-task generalization.

Key Contributions

1. UniACT-dataset. A curated, standardized corpus assembled from six public datasets (OXE, OXE-AugE, AgiBot-Beta, RoboCOIN, RoboMind, and Galaxea), containing more than 6 million trajectories, 9,500+ hours of data, and 20+ robot embodiments, which the authors describe as the largest collection in the non-private domain.

2. The Action Manifold Hypothesis and Action Manifold Learning (AML). The claim that effective robot actions lie on a low-dimensional, smooth manifold governed by physical laws and task constraints. Instead of predicting noise (epsilon-prediction) or velocity (v-prediction), the DiT-based action generator predicts the clean action chunk directly (a-prediction), shifting the learning objective from denoising to projection onto feasible action manifolds.

3. A unified representation and architecture scheme. Delta actions in the end-effector frame, rotation vectors for orientation, and a pad-to-dual-arm strategy that lets one policy network handle both single-arm and dual-arm tasks, plus a dual-stream feature interaction that injects geometry-aware features from optional plug-and-play 3D modules (VGGT and Qwen-Image-Edit) without modifying the backbone.

4. A systematic sampling study and two-stage training paradigm. Multi-level stratified sampling (Trajectory-Uniform, Task-Uniform, Embodiment-Uniform) evaluated for embodiment balance and skill coverage, combined with large-scale pre-training followed by space-aware supervised fine-tuning.

Main Findings

• Benchmark success rates. ABot-M0 achieves average success rates of 98.6% on LIBERO, 80.5% on LIBERO-Plus, 58.3% on RoboCasa GR1 Tabletop Tasks, and 81.2% on Robotwin2.0, outperforming baselines including π0.5, UniVLA, and OpenVLA-OFT.

• Data scale and cleaning. Source trajectories numbered more than seven million across the six datasets; after the multi-stage cleaning pipeline, approximately 16% of trajectories were discarded, leaving the UniACT-dataset with more than six million trajectories in 9,500+ hours and 20+ embodiments.

• Dataset composition is imbalanced. The single-arm dataset OXE-AugE dominates with 67% of total volume, followed by OXE as the second largest contributor. The four dual-arm datasets (AgiBot-Beta, RoboCOIN, RoboMind, Galaxea) collectively account for approximately 17.2%. The authors deliberately reduce AgiBot-Beta's sampling ratio because it features only a single embodiment type.

• Task-Uniform sampling wins on downstream Libero Plus. Table 2 reports total downstream performance of 72.4 for Task-Uniform versus 71.6 for Embodiment-Uniform and 71.3 for Trajectory-Uniform.

• Sampling strategy trade-offs. Trajectory-uniform sampling preserves the original scale imbalance and is dominated by AgiBot-G1, risking homogenization. Embodiment-uniform sampling improves embodiment balance but concentrates probability on a small set of high-frequency skills, slowing coverage growth. Task-uniform sampling yields a Lorenz curve closer to the equality line, a lower Gini coefficient, and faster growth in covered unique skills under the same sampling budget.

• Components are additive. Experiments show the components operate independently, with data standardization, architectural changes, and training redesign producing benefits that accumulate.

• Cross-embodiment validation (partially reported). On the RoboCoin validation set, Task-Uniform and Embodiment-Uniform achieve similar overall MAE, while Task-Uniform is consistently lower on most [the supplied content is truncated here]. Validation sets were built from 1,000 randomly sampled OXE trajectories, 183 AgiBot-Beta tasks (one trajectory each), and 30 RoboCoin trajectories per embodiment (240 total), with MAE as the metric and all models trained for 50k steps.

• Two-stage training preserves prior skills. Supervised fine-tuning with 3D spatial priors substantially improves success rates on insertion, folding, and bimanual door opening while performance on previously learned simple tasks remains stable.

Methodology in Plain English

The authors start from the observation that robot learning data is scattered across incompatible sources. They take six public datasets and run them through a cleaning pipeline that removes empty, garbled, or non-English task prompts (normalizing mixed-language instructions by machine translation), fixes frame/instruction misalignment, decomposes long-horizon tasks into subtask instructions inserted into the language stream, discards visually degraded frames (black images, severe motion blur, heavy occlusions) and ineffective camera viewpoints, filters abnormal action sequences (abnormal lengths, large consecutive deltas, action/video rate mismatches), and drops samples with missing action dimensions or unspecified rotation representations. Everything is converted into LeRobot v2 format.

Then they standardize the action space. All actions become delta actions in the end-effector frame using rotation vectors, where r = θk with r in R³, θ in [0, π], and k a unit vector in R³ with ||k|| = 1. Each arm's action vector is [Δx, Δy, Δz, r, gripper] ∈ R⁷; dual-arm vectors are length 14. To handle single-arm and dual-arm tasks in one network, single-arm action dimensions are zero-padded and treated as right-arm actions within a dual-arm configuration, so the model always outputs 14-dimensional actions but activates only the relevant channels.

The model pairs a VLM backbone (Qwen3-VL) that processes stacked multi-view images and language with an action expert. The action expert is a Diffusion Transformer trained with flow matching, but instead of predicting the flow velocity it predicts the clean action chunk Â_t = [â_t, ..., â_{t+H-1}] of length H. Given ground-truth action A_t, timestep τ ∈ [0,1], and Gaussian noise ε, the noisy action is A^τ_t = τA_t + (1−τ)ε, and the network V_θ takes features φ_t, robot state q_t, and A^τ_t to output Â_t. Although the model predicts actions directly, the loss is computed on velocity, which reduces to a reweighted MSE in action space with weight w(τ) = 1/(1−τ)² — a weight derived from the Jacobian of the action-to-velocity transformation. At inference, actions are generated by solving an ODE from pure noise A⁰_t ~ N(0, I) using a numerical integrator such as Euler.

Training proceeds in two stages: large-scale pre-training on UniACT-dataset (approximately 6 million trajectories) using a discretized action classification loss with a fast token head and dual-weighted sampling based on task category and robot morphology; then supervised fine-tuning of both the VLM and action expert jointly with a small learning rate, with dropout and action noise perturbation, to inject 3D spatial priors for high-precision tasks.

Why This Matters

Impact on research. The paper argues that general embodied intelligence need not emerge from closed, proprietary systems but can develop by aggregating heterogeneous open-source data with shared infrastructure. It provides a concrete recipe for data standardization (end-effector delta actions, rotation vectors, pad-to-dual-arm), a training-objective alternative to noise/velocity prediction, and an empirical study of sampling strategy as a first-class design variable. Releasing the full data processing and training codebase is stated as an explicit goal for reproducible, cumulative progress.

Real-world applications:

• Cross-platform deployment of a single manipulation policy across different robot arms and dual-arm platforms, including transferring skills learned on one embodiment to another.
• High-precision industrial tasks that the paper specifically targets in fine-tuning: insertion, cloth folding, and bimanual door opening.
• Long-horizon household or service manipulation, where subtask decomposition annotations from datasets such as RoboCOIN and RoboMind support complex task planning.
• Spatial-reasoning-heavy tasks (e.g., relative pose between peg and hole, fabric crease directions, inter-arm timing) enabled by the optional 3D perception modules VGGT and Qwen-Image-Edit.

Industry relevance. The work addresses the practical economics of robot data: collection is expensive and hardware-dependent, so aggregating open datasets is a cheaper path than proprietary collection. Unifying single- and dual-arm control in one policy with shared parameters reduces engineering overhead per robot form factor, and the plug-and-play perception module design allows new sensors or skills to be added through fine-tuning without architectural changes.

Future Directions

• Real-robot validation. The reported results are benchmark success rates on LIBERO, LIBERO-Plus, RoboCasa GR1 Tabletop Tasks, and Robotwin2.0. The supplied content does not report physical robot deployments, so sim-to-real transfer remains an open question.
• Richer metadata and modality expansion. The authors state that future VLA datasets should include multiple action representations (joint and end-effector trajectories), fine-grained semantic annotations (task decomposition, key frames, object relationships), sensor and dynamics information (velocities, forces, torques), synchronized multi-view video, and camera intrinsics/extrinsics for auxiliary 3D geometric labels. They also name tactile feedback and force-torque sensing as modalities incorporable through analogous fine-tuning protocols.
• Broader embodiment coverage. The paper calls for datasets spanning single-arm, dual-arm, semi-humanoid, and full humanoid configurations, noting that the challenge escalates as embodiment complexity grows to a full-body humanoid with two dexterous hands and a substantially expanded action space.
• Long-tail generalization. The stratified sampling study shows trade-offs between embodiment coverage and skill coverage; the truncated content stops mid-report on cross-embodiment MAE results, leaving the full picture of when each strategy wins in need of further investigation.

Target Audience

Robotics and embodied-AI researchers building VLA models or large-scale manipulation datasets; machine learning engineers working on diffusion/flow-matching generative policies and action generation; teams curating multi-source robot data who need concrete standards for action representation, coordinate frames, and sampling; and industry practitioners evaluating whether open-source data aggregation can substitute for proprietary robot data collection. Readers without background in flow matching, DiT architectures, or robot action parameterizations will find the paper's technical depth demanding.

Authors’ abstract

Building general-purpose embodied agents across diverse hardware remains a central challenge in robotics, often framed as the ''one-brain, many-forms'' paradigm. Progress is hindered by fragmented data, inconsistent representations, and misaligned training objectives. We present ABot-M0, a framework that builds a systematic data curation pipeline while jointly optimizing model architecture and training strategies, enabling end-to-end transformation of heterogeneous raw data into unified, efficient representations. From six public datasets, we clean, standardize, and balance samples to construct UniACT-dataset, a large-scale dataset with over 6 million trajectories and 9,500 hours of data, covering diverse robot morphologies and task scenarios. Unified pre-training improves knowledge transfer and generalization across platforms and tasks, supporting general-purpose embodied intelligence. To improve action prediction efficiency and stability, we propose the Action Manifold Hypothesis: effective robot actions lie not in the full high-dimensional space but on a low-dimensional, smooth manifold governed by physical laws and task constraints. Based on this, we introduce Action Manifold Learning (AML), which uses a DiT backbone to predict clean, continuous action sequences directly. This shifts learning from denoising to projection onto feasible manifolds, improving decoding speed and policy stability. ABot-M0 supports modular perception via a dual-stream mechanism that integrates VLM semantics with geometric priors and multi-view inputs from plug-and-play 3D modules such as VGGT and Qwen-Image-Edit, enhancing spatial understanding without modifying the backbone and mitigating standard VLM limitations in 3D reasoning. Experiments show components operate independently with additive benefits. We will release all code and pipelines for reproducibility and future research.