MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalization

AI summary

Overview

Research area: Robotics — multi-arm embodied manipulation, vision-language-action (VLA) models, and compositional generalization.

Technical level: Advanced. The paper assumes familiarity with behavior cloning, diffusion/flow-matching policies, and multimodal transformers.

Scope: The paper introduces MA-VLA, a single unified VLA model that decomposes a high-level instruction into per-arm "atomic actions," trains with a role-randomizing trick called Arm Shuffle, and is evaluated on RoboFactory, RoboTwin 2.0, and a real dual-arm SO101 platform for in-domain collaboration and unseen ("compositional") collaboration patterns.

What This Paper Is About

Most VLA models take one high-level language instruction and map it directly to robot control, leaving the division of labor between arms to be inferred implicitly from data. That makes multi-arm collaboration brittle and over-specialized: when the test-time coordination pattern differs from the training patterns, such models fail. The paper's goal is to make cooperation explicit — decomposing a goal into interpretable mid-level atomic actions that are assigned to specific arms — so that a learned system can recombine known skills into collaboration patterns it never saw during training, a capability the authors name multi-arm compositional generalization.

Key Contributions

1. MA-VLA framework. A unified model in which a VLM-based planner decomposes a high-level instruction into a stage-wise sequence of atomic prompts, one per arm (e.g., "grasp object," "hold bowl," "place cube"), and a flow-matching VLA executor grounds those prompts into joint multi-arm control actions in a single forward pass.
2. Arm Shuffle. A training-time permutation that randomly reassigns each arm's state, local view, atomic prompt, and action (with probability p_shuffle), preventing the model from binding behavior to fixed arm identities and enabling recomposition into unseen coordination patterns.
3. View Dropout. A complementary training-time masking of selected camera views (probability p_drop) that encourages redundancy-aware, multi-view perception robust to visual discrepancies.
4. A benchmark for unseen multi-arm collaboration. Evaluation splits in RoboFactory, RoboTwin 2.0, and real-world SO101 where atomic actions remain in-distribution but the collaboration structure — role assignment, ordering, and synchronization — is out-of-distribution. Code, models, and data are released at the linked repository.

Main Findings

• In-domain gains grow with arm count. On RoboFactory (Table 1), MA-VLA averages 83.5% on two-arm tasks (Lift 98.0%, Place 59.0%, Two Arms Stack Cube 86.0%, Pass Shoe 91.0%) versus Pi0 at 80.3%, DP at 27.5%, and Pi0-Fast at 57.5%. On three- and four-arm tasks it averages 83.3% (Stack3 58.0%, Align 96.0%, Deliver 97.0%, Photo 82.0%) versus Pi0 at 76.5%, DP at 15.3%, and Pi0-Fast at 11.8%; the paper states gains grow with more arms.
• Robustness under strong visual disturbance. On RoboTwin 2.0 (Hard), MA-VLA reaches 49.0% average over seven tasks, ahead of Pi0 (41.1%), DP3 (28.6%), Pi0-FAST (27.0%), DP (18.9%), and ACT (12.3%).
• Prior VLAs collapse on unseen collaboration patterns. On the out-of-domain compositional splits (Table 3), DP, Pi0-FAST, and Pi0 each score 0.0% on every column (GBR, RGB, BRG, Pass Two Shoes, Stack Two Bowls; average 0.0%), while MA-VLA averages 13.0% (GBR 28.0%, RGB 9.0%, BRG 9.0%, Pass Two Shoes 9.0%, Stack Two Bowls 10.0%). The paper reports this as improving out-of-domain success by up to 13.0.
• Real-robot role reversal goes from impossible to possible. On the dual-arm SO101 (Table 4, 20 episodes per setting), Pi0 scores 9/20, 12/20, 3/20, and 5/20 in-domain and 0/20 in every out-of-domain setting. MA-VLA scores 12/20, 15/20, 6/20, and 8/20 in-domain, and 10/20, 8/20, 2/20, and 2/20 out-of-domain.
• Each component contributes. Ablation on Three Robots Stack Cube unseen-order (Table 5): no components gives 0.0% out-of-domain / 48.0% in-domain; atomic actions alone 0.0% / 58.0%; atomic actions + Arm Shuffle 7.3% / 52.0%; adding View Dropout 15.3% / 53.0% — a large out-of-domain jump with minimal in-domain degradation.
• Shuffle rate matters. Varying p_shuffle from 0% to 100% raises out-of-domain performance from near-zero to significantly higher levels while in-domain accuracy stays stable with a mild drop.
• One unified model beats three separate ones. A separate per-arm VLA baseline (3 models) reaches 0.0% out-of-domain and 61.0% in-domain (average 30.5%), while MA-VLA (1 model) reaches 15.3% out-of-domain and 53.0% in-domain (average 34.2%) — and separate models carry inference and deployment overhead that grows linearly with arm count.

Methodology in Plain English

Instead of teaching a robot one big command-to-motion mapping, the researchers insert a language step in the middle.

A pretrained vision-language model (GPT-4.1, used only for planning and not fine-tuned) looks at the instruction and images and writes out a short script: for each stage of the task, which arm should do which predefined atomic action, chosen from a fixed vocabulary of prompt templates. Those prompts are concatenated into one string per timestep ("Arm0: ..., Arm1: ..., ...") and fed, along with multi-view images and all arms' joint states, into a single VLA executor. The executor — a Pi0 backbone initialized from the official pi0_base checkpoint with flow-matching and a multi-head projection layer that splits the shared latent into arm-specific action heads — outputs all arms' actions at once.

The crucial training trick is Arm Shuffle: at each iteration, with probability p_shuffle, the bundle of (state, view, prompt, action) belonging to each arm is randomly reassigned to a different arm slot. Because the model can no longer rely on "arm 1 always does X," it must read the atomic prompt to know what to do, which is exactly what lets it handle role assignments it never saw. View Dropout additionally zeroes out some camera views with probability p_drop so the model learns to use redundant spatial cues. Both are data-level perturbations, so the behavior-cloning loss is unchanged.

Evaluation uses 150 expert demonstrations per simulation task (frame-level atomic action labels come from a rule-based parser over contact, grasp status, and object-pose thresholds), 100 rollouts per simulation configuration, and 50 teleoperated demonstrations per real SO101 task with 15,000 training steps, batch size 32, and 20 evaluation episodes. All experiments use two NVIDIA A800 GPUs with batch size 32; RoboFactory two-arm tasks train for 10,000 steps, 3–4 arm tasks for 15,000 steps, RoboTwin 2.0 (Hard) for 30,000 steps, and the horizon is 50 for all tasks. Per-task regularization settings are reported explicitly (for example, p_shuffle=0.8, p_drop=0.2 for Three Robots Stack Cube Reordering; p_shuffle=1.0, p_drop=0.4 for Pass Two Shoes and for SO101).

Why This Matters

Impact on research. The paper reframes multi-arm language conditioning: language is not a single static command but an active coordination channel with explicit per-arm addressability. It also names and benchmarks a capability — recomposing familiar atomic actions into unseen collaboration patterns — that was largely unmeasured in multi-arm VLA work, and shows that a simple, cheap training-time permutation produces non-zero success where strong baselines score exactly zero.

Real-world applications.

• Industrial assembly and manufacturing cells, where 3–4 robot arms must divide tasks and adapt when line layouts or station assignments change.
• Warehouse and logistics automation, including bin picking, packaging, and handover of items between arms with varying roles and orders.
• Household and service robotics such as dual-arm kitchens (stacking bowls, placing cubes, passing objects), where object arrangements and role assignments change constantly.
• Laboratory, surgical-assist, or inspection setups, where a camera-alignment arm and an operating arm must swap or share responsibilities under visual disturbance.

Industry relevance. The result is practical: the model is a single system rather than N per-arm policies, so inference and deployment cost does not grow linearly with arm count, and the regularization requires no new data or reward signal — only a change to how existing demonstrations are fed in. That lowers the barrier to deploying multi-arm fleets that must handle tasks not seen in the training set.

Future Directions

• Close the remaining out-of-domain gap. MA-VLA's compositional success is non-zero but modest (13.0% average in simulation; 2/20 on two real SO101 out-of-domain tasks), so scaling compositional robustness — and understanding what limits it — is the obvious next step.
• Scale to more arms and longer horizons. Evaluation covers 2–4 arms with a horizon of 50; whether Arm Shuffle stays effective for larger teams and extended task pipelines is not reported.
• Reduce planner dependence. The planner is an un-fine-tuned GPT-4.1 over a fixed prompt vocabulary; whether the atomic-action library can be learned or expanded automatically, and whether an open planner suffices, remains open.
• Generalize the guarantee. The formalization assumes test-time atomic actions are in-distribution (U_test = U_train); extending to genuinely novel skills, failure recovery, and closed-loop replanning when a subgoal fails is untested.

Target Audience

Robotics and embodied-AI researchers working on VLA models, multi-arm/bimanual manipulation, and imitation learning; graduate students entering multi-arm coordination; and robotics engineers evaluating whether per-arm policies or a unified instruction-conditioned model better suits deployable multi-arm systems. Readers should be comfortable with behavior cloning, flow-matching/diffusion policies, and standard success-rate benchmarking.

Authors’ abstract

Multi-arm collaboration is becoming a core capability in embodied manipulation. Recent vision-language-action (VLA) models integrate perception, language, and control, but most represent language as a single global instruction and do not provide an explicit mechanism for assigning and composing arm-specific behaviors. This design limits transfer to collaboration patterns that differ from those observed during training. We present MA-VLA, a unified framework for multi-arm collaboration via atomic action assignment. MA-VLA decomposes cooperative behavior into mid-level atomic prompts and allocates them to individual arms, enabling explicit subgoal specification and compositional reuse across tasks. To reduce reliance on fixed execution roles, we introduce Arm Shuffle, a training-time permutation of the observation, state, and assigned atomic prompts for each arm. This permutation enforces role-agnostic instruction following and supports recomposition into unseen coordination patterns, which we term multi-arm compositional generalization. We also construct a benchmark in which test-time collaboration patterns are absent in training set. Across simulation and real-world evaluations, prior state-of-the-art VLAs largely fail under these unseen collaborations, while MA-VLA consistently succeeds. These results indicate that structured, per-arm atomic action assignment offers a practical route to scalable generalization in multi-arm embodied systems. Code, models, and data are available at https://github.com/zhangzaibin/future-robots