Dexbotic: Open-Source Vision-Language-Action Toolbox

AI summary

Overview

Research area: Robotics / Embodied AI — specifically Vision-Language-Action (VLA) models, the class of models that map camera observations and language instructions directly to robot actions.

Technical level: Advanced. The paper assumes familiarity with VLMs, LLM training pipelines, diffusion and flow-matching action heads, and robot learning benchmarks. It is a software-engineering and infrastructure paper rather than a single algorithmic result.

One-sentence scope: Dexbotic is an open-source PyTorch toolbox that unifies the architecture, data format, training and inference path for a family of existing VLA policies, and ships stronger pretrained checkpoints that improve those policies' success rates across simulation and real-world tasks.

What This Paper Is About

VLA research is fragmented: different labs use different frameworks, codebases, data formats and base models, so comparing two policies requires setting up several environments and re-implementing data loaders. On top of that, many popular VLA policies are built on outdated language backbones, so users do not benefit from recent advances in LLMs. Dexbotic's goal is to be a single codebase where mainstream VLA policies can be reproduced with one environment setup, run from an experiment-centric configuration script, and initialized from stronger pretrained foundation models supplied by the authors.

Key Contributions

1. A unified modular VLA framework. Dexbotic decomposes every VLA policy into a vision-language model (VLM) part (vision encoder, projector, LLM) and an action expert part (diffusion transformer, MLP, or mixture-of-experts coupled with the LLM), and integrates multiple mainstream policies — π₀, OpenVLA-OFT, CogACT, MemoryVLA and MUVLA — under a single PyTorch codebase.

2. New pretrained foundation models. The authors pretrain DexboticVLM from scratch using CLIP as the vision encoder, a two-layer MLP projector and Qwen2.5 as the LLM, following a LLaVA-style pipeline (freeze vision encoder and LLM, train projector for cross-modal alignment, then update the whole network) on data from LLaVA and Cambrian. On top of it they release Dexbotic-Base (discrete), Dexbotic-CogACT (single-arm continuous) and a hybrid-arm continuous model.

3. An experiment-centric development framework. Instead of YAML configuration, experiments inherit a base_exp script and override only the fields that differ, so users can fork a new set of hyperparameters without copying a whole file, add models or tasks, and launch training or inference with a single script invocation.

4. The Dexdata format plus cloud/local training and deployment. Dexdata stores each episode as an mp4 video plus a jsonl file of per-frame multi-view images, robot state, prompt and metadata, with an auto-generated index_cache.json. The authors claim it saves storage relative to the LeRobot and RLDS formats, and the toolbox runs on cloud platforms such as Alibaba Cloud and Volcano Engine as well as consumer GPUs such as the RTX 4090.

Main Findings

• SimplerEnv (WidowX, Visual Matching suite): DB-CogACT reaches 69.5 average success versus CogACT's 51.3 (+18.2); DB-OFT reaches 76.4 versus OpenVLA-OFT's 30.2 (+46.2); DB-MemVLA reaches 84.4 versus MemoryVLA's 71.9 (+12.5). The four tasks are Spoon on Towel, Carrot on Plate, Stack Cube and Eggplant in Basket.

• CALVIN (ABC→D split, 1000 rollouts per task): DB-CogACT reaches 4.06 average length versus CogACT's 3.25 (+0.81), and DB-OFT reaches 3.54 versus OpenVLA-OFT's 3.47 (+0.07).

• RoboTwin2.0 (four dual-arm tasks): DB-CogACT reaches 58.5 average success versus CogACT's 43.75 (+14.75) on Adjust Bottle, Grab Roller, Place Empty Cup and Place Phone Stand.

• LIBERO: Gains are small because performance is described as nearly saturated. DB-CogACT reaches 94.9 average versus CogACT's 93.6 (+1.3), and DB-MemVLA reaches 97.0 versus MemoryVLA's 96.7 (+0.3).

• ManiSkill2 (five pick-and-place tasks): DB-CogACT reaches 58 average success versus CogACT's 40 (+18.0), and DB-OFT reaches 63 versus OpenVLA-OFT's 21 (+42.0), with the original OFT described as producing "undesirable performance."

• Continuous-action modeling uses 256 bins per action dimension, and the discrete pretrained model predicts N tokens where N is the degree of freedom.

• Hybrid-arm support required an architectural change: CogACT's noise tokens were increased from 7 to 16 to cover both 6-DoF and 7-DoF arms, with the first half representing the left arm and the second half the right arm; for single-arm data the second half's loss is ignored.

• Real-world tasks: Data was collected by teleoperation on UR5e, ALOHA, ARX5 and Franka, with 500–1000 demonstrations per task depending on difficulty. Dexbotic achieves 100% success on "set the plates" and 80% on "search the green box", while fine-grained tasks such as "Shred the scrap paper" and "Pour fries into plate" remain challenging. MemoryVLA is shown to solve long-horizon, memory-requiring tasks such as "Push buttons sequentially".

• DOS-Twins (Dexbotic Open Source-Twins) is a Real2Sim2Real simulator in the Dexbotic ecosystem, using Isaac Sim as the backend physics engine and 3D Gaussian Splatting (3DGS) as the rendering frontend. It targets three forms of consistency: visual, motion and interaction, with geometric structure errors for objects kept within a millimeter.

Methodology in Plain English

The authors treat VLA design as a modular decomposition problem. Every policy is split into a "brain" (VLM: vision encoder + projector + LLM) and a "hand" (action expert). The brain turns images and text into multimodal tokens; the hand turns those tokens into action chunks, either by decoding discrete tokens directly (each action dimension quantized into 256 bins) or by running a diffusion or flow-matching process.

They then pretrain a new brain, DexboticVLM, from scratch by pairing CLIP with Qwen2.5 through a two-layer MLP projector, training the projector first and the whole network afterward. This brain becomes the initialization for downstream policies. A discrete checkpoint called Dexbotic-Base is trained on a mix of Open-X Embodiment data, simulation data from RLBench, LIBERO and ManiSkill2, and real UR5 data. From that, a continuous CogACT-style model is trained on 52 private manipulation tasks gathered with eight different single-arm robots, and then further trained for two-arm settings by expanding the noise-token count and adding data such as Robomind, AgiBot World and private ALOHA data.

To evaluate, they swap each official policy's initialization for their pretrained checkpoints and rerun the same benchmark suites, reporting success rates side by side with the official numbers. For real-world validation, they collect teleoperated demonstrations on four robot platforms, convert them to Dexdata, fine-tune the pretrained models, and report task success. For scalable evaluation, they build DOS-Twins, a simulator that reconstructs real setups using 3D scanning and Gaussian Splatting so that policies trained on real or public data can be evaluated in simulation with matched camera viewpoints, kinematics and gripper geometry.

Why This Matters

Impact on research: The paper's argument is that the field cannot compare policies fairly when every policy requires its own environment, data format and base model. By fixing the architecture split and giving every supported policy the same stronger initialization, it makes comparisons more attributable to the policy rather than to the pretrained backbone. The reported jumps — for example +42.0 on ManiSkill2 for OFT — suggest that much of the apparent gap between policies may actually be a gap in pretrained representations.

Real-world applications:

• Tabletop and kitchen manipulation on single-arm platforms such as UR5, Franka and ARX5.
• Dual-arm manipulation on ALOHA-style hardware, supported by the hybrid-arm pretrained model.
• Long-horizon tasks that need memory, such as pushing buttons in sequence.
• Warehouse-style pick-and-place: grasping specified objects and placing them at marked 3D positions, as tested by ManiSkill2.

Industry relevance: The toolbox explicitly targets both university and enterprise users, supporting large-scale cloud training on Alibaba Cloud and Volcano Engine as well as local training on an RTX 4090. A unified data format plus an inference service (a DexClient talking over HTTP to a Flask-based Web API) lowers the barrier to deploying VLA models on real hardware, and unified benchmarks give buyers and builders a like-for-like basis for choosing a policy.

Future Directions

• Broader policy coverage. The authors state that π₀.₅, and navigation policies NaVid and NaVILA, will be supported in the near future, alongside existing integration for whole-body control interfaces.
• More robot platforms. Dexbotic currently targets UR5, Franka and ALOHA; the paper says more mainstream robotic platforms will be added.
• Closing the fine-grained manipulation gap. Real-world results show high success on coarse tasks but difficulty on tasks such as shredding paper or pouring fries, leaving open how to address dexterous, fine-grained manipulation.
• Scaling Real2Sim evaluation. DOS-Twins is presented as an evaluation pipeline for public real-world datasets; an open question is whether it can become a shared standard for the proposed RoboChallenge comparisons without losing sim-to-real fidelity.

Target Audience

Robotics and embodied-AI researchers who want to reproduce or build on VLA policies without rebuilding every environment; engineers deploying manipulation policies on real hardware who need a unified data format, training pipeline and inference service; and students or smaller labs that need to run experiments on consumer GPUs such as an RTX 4090 rather than large clusters.

Authors’ abstract

In this paper, we present Dexbotic, an open-source Vision-Language-Action (VLA) model toolbox based on PyTorch. It aims to provide a one-stop VLA research service for professionals in the field of embodied intelligence. It offers a codebase that supports multiple mainstream VLA policies simultaneously, allowing users to reproduce various VLA methods with just a single environment setup. The toolbox is experiment-centric, where the users can quickly develop new VLA experiments by simply modifying the Exp script. Moreover, we provide much stronger pretrained models to achieve great performance improvements for state-of-the-art VLA policies. Dexbotic will continuously update to include more of the latest pre-trained foundation models and cutting-edge VLA models in the industry.