MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agent

AI summary

Overview

• Research area: Robotics, specifically Vision-Language-Action (VLA) models and data-free model merging for multi-skill robot policies.
• Technical level: Advanced. The paper assumes familiarity with LoRA fine-tuning, transformer attention variants, singular value decomposition, and model-merging methods such as Task Arithmetic and TIES.
• Scope in one sentence: The paper diagnoses why independently fine-tuned VLA experts cannot be merged into one policy, redesigns the VLA architecture to be mergeable by construction, and validates the resulting model on LIBERO, LIBERO-Plus, RoboTwin, and a real SO101 robotic arm.

What This Paper Is About

VLA models perform well when fine-tuned for a single task or embodiment, but combining several such experts into one generalist policy fails badly: merging them yields near-zero success rates. This paper asks what property of VLA fine-tuning causes that failure, and answers it by building MergeVLA, an architecture whose components stay mergeable by design, together with a training-free router that infers the task at test time when the task identity is unknown.

Key Contributions

1. An empirical diagnosis of VLA non-mergeability. Through a decomposition of learnable parameters, the authors identify two failure modes: LoRA adapters in the VLM backbone diverge into task-specific directions, and train-from-scratch action experts accumulate inter-block dependencies through self-attention feedback that spreads task information across layers.
2. A merge-oriented VLA architecture. MergeVLA applies sparsely activated LoRA adapters through task masks in the VLM, and redesigns the action expert to use cross-attention-only blocks (removing self-attention) with a sigmoid gate replacing the tanh gate.
3. A specialization hierarchy for merging. Shallow action-expert blocks are merged by simple weight averaging, while the deeper blocks forming the "expert head" are kept unmerged per task because their parameters are too task-specific.
4. A training-free test-time task router. Given an unknown task, the router runs each masked VLM variant, projects hidden states onto the principal value subspaces of the merged action expert, scores task relevance, and selects the highest-scoring task mask and expert head from the initial observation alone.

Main Findings

• Direct merging of VLA experts collapses. Applying Task Arithmetic to VLA-Adapter on LIBERO produces 0.0% success across Spatial, Object, Goal, and Long suites; adding task masks still yields 0.0% unless the final action block is excluded.
• Even single-task experts do not transfer. Evaluating single-task fine-tuned models on unseen task suites gives 0% success for all methods tested, indicating a lack of cross-skill generalization.
• Parameter selfishness is extreme. When merging only four tasks, the proportion of parameters retained by exactly one task mask exceeds 75% for both TA and TIES, and the selfish ratio rises to around 75% as the number of merged tasks increases from 2 to 4.
• Merged LIBERO performance approaches fine-tuning. MergeVLA with TIES reaches a 90.2% average success rate (94.8 Spatial, 94.6 Object, 91.8 Goal, 79.4 Long), 6.5% below the single-task fine-tuned MergeVLA average of 96.7%. Other combinations reach 89.9% (WUDI), 89.7% (TA), 88.0% (KnOTS), 81.6% (TSV), and 65.5% (EMR).
• Architectural changes alone improve robustness. The removal of self-attention and the switch to a sigmoid gate produce a 13.4% higher success rate than VLA-Adapter under varying corruptions on LIBERO-Plus.
• LIBERO-Plus robustness. Single-task MergeVLA averages 72.4% across the seven shifts, above VLA-Adapter (59.0%), pi0 (56.3%), and OpenVLA (16.3%). Merged MergeVLA reaches 62.5% with TIES, 61.6% with TA, and 53.5% with TSV, compared with 10.8% for merged VLA-Adapter.
• Cross-embodiment merging works with a larger expert head. On RoboTwin Setting A (three embodiments, one task), MergeVLA with TIES routing the last two blocks averages 88.7%, close to the 88.0% single-task baseline. In Setting B (three embodiments, three tasks), routing the last three blocks with TIES reaches 70.7%, versus 48.7% for TA with the last block, 49.3% for TA with the last two blocks, and 59.3% for TIES with the last block.
• Real-robot results. On the SO101 arm, TIES-based merging averages 90.0% (90.0 Pick & Place, 90.0 Push Cube, 90.0 Stack Cube), matching the 90.0% single-task fine-tuning average, while TA-based merging averages 66.7%.
• Task routing needs one step. A single routing step using the initial observation at t=0 is sufficient to identify the correct task; the selected mask and expert head are then fixed for the rest of the episode.

Methodology in Plain English

The authors first look inside fine-tuned VLA models to see which parameters change and how. They find that different tasks push the same LoRA adapters in different directions, and that action experts trained from scratch develop cross-layer coupling through self-attention, so task-specific information bleeds across blocks. Both effects make the models impossible to average together.

Their response is to change the architecture before training rather than fix merging afterward. In the VLM, they build masks per task that retain only the merged parameters whose task-specific update is strong and consistent with the overall merge direction, so each task activates its own slice of a shared merged weight set. In the action expert, they delete self-attention layers so blocks no longer feed back into one another, and replace the tanh gate with a sigmoid gate so the pretrained VLM signal is always preserved rather than suppressed by negative activations. This keeps most blocks composable and mergeable by plain averaging, except the deepest block or blocks, which stay separate per task as the "expert head."

When the task is unknown, the model needs to pick the right mask and head. The router takes the value projection matrices of a cross-attention block, extracts their top principal directions via SVD, and measures how strongly each masked VLM's hidden states activate those directions. The strongest average response across the task and action paths wins, and softmax over the scores gives the selection probability. No extra training or labels are required.

Why This Matters

For research, the paper reframes model merging for VLAs as an architecture-design problem rather than a post-hoc weight-manipulation problem, and it provides a decomposition-based explanation of why VLA experts resist merging when LLM and VLM experts generally do not.

Real-world applications include:

• Household and service robots that must perform pick-and-place, pushing, and stacking without training a single joint policy over all demonstrations.
• Multi-embodiment fleets, where arms with different morphologies (as tested with Aloha-Agilex, ARX-X5, and Piper in RoboTwin) reuse a shared merged backbone with per-embodiment expert heads.
• Industrial manipulation cells where new skills are added by training a lightweight expert and merging it in, rather than retraining the whole system.
• Robustness-critical deployment, where visual and language shifts such as background texture, camera viewpoint, and lighting changes occur (the LIBERO-Plus perturbations S1 through S7).

Industry relevance comes from the cost profile: the merged system uses 0.70B parameters to cover all four LIBERO tasks versus 0.68B times 4 for separate experts, and the merging procedure needs no access to the original training datasets and no joint retraining.

Future Directions

• Deciding automatically how many deep blocks must remain unmerged, since the paper fixes this manually per setting (last block for LIBERO and real-world, last two blocks for RoboTwin Setting A, last three for Setting B).
• Improving the router for cross-embodiment settings, where morphology and action-space differences create stronger conflicts than the single-embodiment case.
• Scaling the study beyond four merged tasks: the selfish-ratio analysis only covers merging 2 to 4 tasks, and behavior at larger skill counts is not reported.
• Extending evaluation to more diverse real-world tasks and lighting or camera conditions, since the physical-robot evaluation covers three cube manipulation tasks with 20 rollouts each.

Target Audience

Robotics and embodied-AI researchers working on VLA policies, parameter-efficient fine-tuning, and model merging; engineers building multi-skill robot systems who want to consolidate separately trained experts without joint retraining; and graduate students interested in the architectural conditions that make neural network weights composable. Readers need background in transformer attention and LoRA to follow the technical sections.

Note on completeness: the supplied paper content is truncated inside the ablation study on mask ratio, so the ablation's final success-rate numbers and the reported hyperparameter appendix are not fully available in this text; the setup values reported are k_r = 8, mask ratio lambda = 0.6, merging scaling factor alpha = 1, backbone Qwen2.5-0.5B, and fine-tuning on a single NVIDIA A6000 Ada GPU (48 GB).

Authors’ abstract

Recent Vision-Language-Action (VLA) models reformulate vision-language models by tuning them with millions of robotic demonstrations. While they perform well when fine-tuned for a single embodiment or task family, extending them to multi-skill settings remains challenging: directly merging VLA experts trained on different tasks results in near-zero success rates. This raises a fundamental question: what prevents VLAs from mastering multiple skills within one model? With an empirical decomposition of learnable parameters during VLA fine-tuning, we identify two key sources of non-mergeability: (1) Finetuning drives LoRA adapters in the VLM backbone toward divergent, task-specific directions beyond the capacity of existing merging methods to unify. (2) Action experts develop inter-block dependencies through self-attention feedback, causing task information to spread across layers and preventing modular recombination. To address these challenges, we present MergeVLA, a merging-oriented VLA architecture that preserves mergeability by design. MergeVLA introduces sparsely activated LoRA adapters via task masks to retain consistent parameters and reduce irreconcilable conflicts in the VLM. Its action expert replaces self-attention with cross-attention-only blocks to keep specialization localized and composable. When the task is unknown, it uses a test-time task router to adaptively select the appropriate task mask and expert head from the initial observation, enabling unsupervised task inference. Across LIBERO, LIBERO-Plus, RoboTwin, and multi-task experiments on the real SO101 robotic arm, MergeVLA achieves performance comparable to or even exceeding individually finetuned experts, demonstrating robust generalization across tasks, embodiments, and environments. Project page: https://mergevla.github.io/