ACE-Brain-0: Spatial Intelligence as a Shared Scaffold for Universal Embodiments

AI summary

Overview

Research area: Robotics / embodied AI — specifically multimodal large language models (MLLMs) for cross-embodiment agent intelligence.

Technical level: Intermediate. The high-level idea is intuitive, but the training pipeline (expert merging, GRPO reinforcement tuning) assumes familiarity with MLLM training and model-merging methods.

Scope: This paper proposes and evaluates ACE-Brain-0, an 8B-parameter generalist "foundation brain" that unifies spatial reasoning, autonomous driving, low-altitude UAV sensing, and embodied manipulation inside a single model, using spatial cognition as the shared substrate across embodiments.

What This Paper Is About

Building one AI model that can drive a car, fly a drone, and control a robot arm is hard because these domains have conflicting training objectives, wildly imbalanced data, and no shared action space. The standard fixes—training on everything at once, or fine-tuning on domains one after another—either cause tasks to interfere with each other or cause the model to forget what it learned earlier. ACE-Brain-0's claim is that 3D spatial understanding is the one capability all these embodiments genuinely share, so it can serve as a common foundation that makes stable multi-domain training possible.

Key Contributions

1. Spatial intelligence as a shared cross-embodiment scaffold. The authors argue and empirically show that a spatial-reasoning checkpoint transfers productively into driving, UAV, and manipulation experts—not just into spatial benchmarks.

2. The Scaffold-Specialize-Reconcile (SSR) training paradigm. A five-stage pipeline that first builds a shared spatial foundation, then trains separate domain experts in isolation, then fuses them without any training data via model merging, followed by embodied fine-tuning and optional GRPO reinforcement learning.

3. ACE-Brain-0 itself. An 8B open-weight generalist embodied brain built on Qwen3-VL, released with code, weights (Hugging Face), and a project page.

4. Broad empirical validation. Evaluation across 24 spatial and embodiment-related benchmarks, with head-to-head comparisons against GPT-4o, Gemini-2.5-Pro, Claude-4-Sonnet, and existing embodied brains such as RoboBrain2.5-8B, VeBrain-7B, and MiMo-Embodied-7B.

Main Findings

• State-of-the-art spatial reasoning at 8B scale. ACE-Brain-0 reaches 92.0% on SAT (vs. 79.3% for Gemini-2.5-Pro) and 82.1% on MindCube (vs. 57.6% for Gemini-2.5-Pro and 34.6% for Vlaser-8B), with 63.3% on VSI and 83.9% on BLINK.

• Large driving gains. 71.2% on MME-RealWorld and 91.7% on NuPlanQA, beating Gemini-2.5-Pro (67.0%, no NuPlanQA result) and the strongest embodied baseline in most categories. Also 77.8% on MAPLM and 58.8% on NuscenesQA versus GPT-4o's 34.3%.

• New records on low-altitude benchmarks. 56.9% on UrbanVideo-Bench and 70.3% on AircopBench, with a striking 61.2% on HRVQA (compared to 37.6% for InternVL3-8B). Avi-Math is a narrower win at 35.0%.

• Competitive embodied results. 64.6% on RoboVQA (a large margin over GPT-4o's 34.5% and MiMo-Embodied-7B's 32.8%), 70.0% on OpenEQA, 77.3% on EmbSpatial-Bench, 55.3% on EgoPlan-Bench2, and 42.3% on EB-Habitat.

• The ablation is the strongest evidence for the central thesis. Training a driving expert directly from the base model yields +11.1%; starting from the spatial scaffold yields +25.6%. For UAV, +11.0% becomes +16.5%. For embodied tasks, direct fine-tuning actually hurts performance (-1.9%), but the spatial-initialized route gains +5.4%. This shows the scaffold converts an unproductive domain into a productive one.

• Merging beats joint and sequential training. The authors report that SSR avoids the catastrophic forgetting seen in sequential fine-tuning and exceeds what joint mixed-data training achieves, framing the result as a resolution of the stability–plasticity dilemma.

Methodology in Plain English

The approach has five sequential stages:

1. Scaffold. Start from Qwen3-VL-8B, do general instruction tuning to activate baseline abilities, then train heavily on spatial reasoning data. This produces a "spatial expert" that becomes the shared starting point for everything else.

2. Specialize. From that spatial checkpoint, train separate experts for autonomous driving and UAVs on their own data, in complete isolation. Because they never see each other's gradients, there is no interference—but also no unified model yet.

3. Reconcile. Fuse the experts into a single model without any training data. The method works with task vectors (the difference between each expert's weights and the base weights) and solves an optimization problem, initialized as the average of all experts, that minimizes the squared difference between each expert's output and the merged model's output on its own layer. This runs for 1,000 Adam iterations at a learning rate of 1e-5 using the FusionBench framework. The authors also tried SVD-based task singular vector merging and simple weight averaging (Model Soups).

4. Embodied SFT. Fine-tune the merged model on egocentric embodied data covering interaction, task planning, and action prediction.

5. GRPO. Optional reinforcement fine-tuning on 100k mixed samples, using Group Relative Policy Optimization. For each question, the model samples multiple answers, a reward model scores them, and the scores are normalized within the group to form advantages. Notably, they drop the KL penalty against a reference policy because the clipped objective alone was stable enough.

Two engineering details worth noting: visual tokens are conceptually organized into five domain categories (General, Spatial, Driving, Aerial, Embodied) and concatenated with text tokens for the decoder, and the loss uses "square averaging" rather than naive token or sample averaging to avoid bias toward longer or shorter responses.

Why This Matters

Impact on research. The paper reframes cross-embodiment learning from a data-volume problem into a structure problem: find the shared cognitive primitive, build it first, branch out, then recombine. The ablation table is the kind of clean causal evidence that is often missing in multi-domain training papers, since it isolates initialization source as the only variable. It also pushes data-free model merging—largely a multi-task vision technique—into embodied AI as a practical alternative to joint training.

Real-world applications:

• Autonomous vehicles that inherit strong spatial priors from a shared brain rather than training a narrow pipeline per task, potentially reducing cost and improving generalization to rare traffic scenarios.
• Drone delivery and urban air mobility, where the same model handles aerial localization, safety-critical scene reasoning, and bird's-eye traffic understanding.
• Household and warehouse robots that need fine-grained manipulation plus high-level task planning from the same weights.
• Multi-robot or mixed-fleet coordination, where a vehicle, a drone, and a ground robot could share a representation and reasoning substrate instead of running incompatible stacks.

Industry relevance. The model is 8B parameters and openly released, which is a practical operating point for edge deployment on vehicle or robot compute. The SSR recipe offers a concrete cost structure: train a scaffold once, then add new embodiments as cheap independent experts that get merged in. That is directly relevant to any organization that wants to add a new robot form factor without retraining the whole system or accepting forgetting.

Future Directions

• Extending the scaffold to modalities beyond vision. The paper covers images and video, but touch, proprioception, and force feedback are absent—yet critical for manipulation. Whether spatial cognition remains the right shared primitive when haptics enter is an open question.

• Generalizing the merge to more and more dissimilar embodiments. With three expert branches the reconciliation is tractable. Behavior under dozens of experts with conflicting task vectors, and whether interference bounds still hold, is untested.

• Closing the embodied gap. Embodied manipulation still lags relative to the dramatic gains in spatial and driving benchmarks, and even the spatial-scaffold route only adds +5.4%. Why fine-grained action understanding resists transfer from spatial priors deserves a dedicated investigation.

• Whether the scaffold is truly domain-agnostic or just a good warm start. The ablations show spatial initialization helps, but do not fully separate "spatial reasoning is the shared primitive" from "any large, well-curated intermediate checkpoint helps." A controlled comparison against non-spatial intermediate checkpoints of matched scale would sharpen the claim.

Target Audience

Researchers and engineers working on embodied AI, vision-language-action models, and multimodal foundation models, particularly those interested in multi-domain training without catastrophic forgetting. Also relevant to practitioners in autonomous driving, UAV autonomy, and robotics who want a single model serving multiple platforms. Readers with only a general ML background can follow the motivation and results, but the merging math and GRPO formulation require prior exposure to parameter-efficient adaptation and RL fine-tuning.

Authors’ abstract

Universal embodied intelligence demands robust generalization across heterogeneous embodiments, such as autonomous driving, robotics, and unmanned aerial vehicles (UAVs). However, existing embodied brain in training a unified model over diverse embodiments frequently triggers long-tail data, gradient interference, and catastrophic forgetting, making it notoriously difficult to balance universal generalization with domain-specific proficiency. In this report, we introduce ACE-Brain-0, a generalist foundation brain that unifies spatial reasoning, autonomous driving, and embodied manipulation within a single multimodal large language model~(MLLM). Our key insight is that spatial intelligence serves as a universal scaffold across diverse physical embodiments: although vehicles, robots, and UAVs differ drastically in morphology, they share a common need for modeling 3D mental space, making spatial cognition a natural, domain-agnostic foundation for cross-embodiment transfer. Building on this insight, we propose the Scaffold-Specialize-Reconcile~(SSR) paradigm, which first establishes a shared spatial foundation, then cultivates domain-specialized experts, and finally harmonizes them through data-free model merging. Furthermore, we adopt Group Relative Policy Optimization~(GRPO) to strengthen the model's comprehensive capability. Extensive experiments demonstrate that ACE-Brain-0 achieves competitive and even state-of-the-art performance across 24 spatial and embodiment-related benchmarks.