MiMo-Embodied: X-Embodied Foundation Model Technical Report

AI summary

Overview

Research area: Robotics / Embodied AI / Autonomous Driving — multimodal vision-language foundation models for physical-world agents.

Technical level: Advanced (assumes familiarity with VLMs, chain-of-thought reasoning, reinforcement learning from group-relative preferences, and benchmark evaluation methodology).

Scope: This technical report describes MiMo-Embodied, a single open-source 7B vision-language model that unifies autonomous driving and embodied AI tasks and claims state-of-the-art performance across 29 benchmarks spanning both domains.

What This Paper Is About

Specialized embodied VLMs today exist in two disconnected camps — indoor robots (task planning, affordance, spatial reasoning) and outdoor autonomous vehicles (perception, prediction, planning) — with a substantial domain gap that prevents cross-domain generalization. MiMo-Embodied argues that these two domains actually share underlying spatial, temporal, and causal reasoning structures, and that training a single model on both produces positive transfer rather than interference. The paper's goal is to build, train, and comprehensively evaluate the first unified foundation model that bridges both domains at state-of-the-art level.

Key Contributions

1. First unified cross-embodied foundation model. MiMo-Embodied is a single 7B VLM (built on MiMo-VL) that handles both autonomous driving and embodied AI, with shared vision encoder, projector, and LLM backbone rather than task-specific heads or separate models.

2. A four-stage progressive training recipe that demonstrates positive cross-domain transfer. The stages move from general + embodied supervision, to autonomous driving specialization, to chain-of-thought fine-tuning, to GRPO-based reinforcement learning — showing the two domains reinforce rather than conflict with each other.

3. A curated multi-domain dataset spanning three categories. General multimodal understanding (grounding, documents, video, multimodal reasoning), embodied AI (affordance prediction, high-level planning, spatial understanding), and autonomous driving (environmental perception, status prediction, driving planning) — assembled from over 30 existing datasets and refined for training.

4. Comprehensive cross-embodiment evaluation. State-of-the-art on 17 embodied AI benchmarks across three capability families, and on 12 autonomous driving benchmarks across three capability families, beating open-source, closed-source (GPT-4o, Claude-Sonnet-4, Gemini2.5-Pro, Qwen-VL-Max), and specialized embodied baselines.

Main Findings

• Affordance prediction dominance: MiMo-Embodied reaches 82.30 on RoboRefIt (vs 80.42 for Qwen2.5-VL, 70.40 for RoboBrain-2.0), 46.93 on VABench-Point, 65.50 on Part-Afford, and 69.81 on RoboAfford-Eval — the strongest results across all five affordance benchmarks tested.

• Specialized embodied models fall short: Baselines like VeBrain (1.67 on VABench-Point, 2.08 on RoboAfford-Eval), Magma (4.95 on RoboRefIt), and Cosmos-Reason1 (5.96 on VABench-Point) score far below both general-purpose VLMs and MiMo-Embodied, indicating that narrow specialization does not imply strong general embodied competence.

• Planning and spatial gains: MiMo-Embodied also leads on EgoPlan2 (43.00), RoboVQA (61.99), and Cosmos planning (56.80), edging out Qwen2.5-VL on planning tasks where closed-source frontier models are weaker (e.g., Claude-Sonnet-4 at 8.51 on RoboVQA).

• Cross-domain positive transfer is the central empirical claim. The authors argue that multi-stage learning, curated data construction, and CoT/RL fine-tuning cause autonomous driving and embodied AI capabilities to mutually reinforce, rather than compete for model capacity.

• Large margins on fine-grained affordance tasks specifically. The gap is most pronounced on benchmarks requiring precise spatial grounding (points, bounding boxes, part-level affordances), suggesting the training data mixture meaningfully improved physical grounding rather than just caption-level understanding.

• Reinforcement learning improves precision, not just correctness. The GRPO stage uses task-specific reward signals — exact answer matching for multiple choice, IoU for spatial grounding/pointing, and template checks for format compliance — which the authors credit with sharpened output reliability on corner cases.

Methodology in Plain English

The authors start from an existing strong vision-language model (MiMo-VL) and progressively teach it to handle physical-world tasks through four sequential fine-tuning stages.

Stage 1 blends general multimodal data with embodied AI data (affordance prediction, task planning, spatial understanding) so the model learns to ground language in physical scenes while retaining broad visual recognition. Stage 2 adds autonomous driving data on top, including multi-camera inputs, temporal video sequences, and safety-critical perception tasks like hazard identification and intention prediction — this is where the cross-domain mixing happens. Stage 3 injects chain-of-thought training samples where the model is shown explicit reasoning traces: analyze context, propose options, evaluate, justify. Stage 4 applies reinforcement learning via GRPO, where the model generates multiple candidate answers per query and is rewarded based on task-appropriate signals (exact match, spatial IoU, format compliance).

Architecturally, the model is a conventional VLM stack — a Vision Transformer encodes images/video, an MLP projector maps visual tokens into the LLM's latent space, and the LLM produces textual reasoning and answers. All components stay trainable in every stage, with a long 32,768-token context (helpful for video and multi-frame driving sequences). Training uses AdamW, cosine learning rate scheduling, batch size 512 for stages 1–3, and batch size 32 with a lower learning rate for the RL stage.

Why This Matters

Research impact: The paper challenges the assumption that embodied AI and autonomous driving require separate models due to domain gap. If cross-domain transfer is real and reproducible, it suggests a consolidation path for embodied foundation models — one backbone, many physical embodiments — rather than a proliferation of narrow specialists.

Real-world applications:

• Robotics manipulation: Fine-grained affordance grounding enables robots to identify graspable parts, handles, and placeable regions from natural language commands.
• Autonomous driving assistants: Unified perception, intent prediction, and interpretable driving reasoning support driver-assist and explainable autonomy systems.
• Shared agent architectures: A single model deployed across factory robots, delivery vehicles, and indoor service robots could cut engineering and deployment costs.
• Human-robot interaction: Spatial understanding and first-person planning capabilities support instruction-following in cluttered, realistic environments.

Industry relevance: The release is open-source (Xiaomi Embodied Intelligence Team), directly competing with proprietary frontier models (GPT-4o, Gemini2.5-Pro, Claude-Sonnet-4) that often underperform on embodied benchmarks. That positions MiMo-Embodied as a practical, deployable alternative for robotics and automotive companies unwilling to depend on closed APIs — and demonstrates that a 7B model can beat much larger generalist systems when trained with domain-appropriate data and objectives.

Future Directions

• Verifying transfer causality: The paper asserts mutual reinforcement between domains, but the exact mechanism (shared spatial priors? shared temporal reasoning? dataset overlap?) is not isolated — ablation studies separating domain contributions would clarify this.
• Scaling and embodiment breadth: Whether the four-stage recipe and positive transfer hold at larger model sizes, and whether it extends to additional embodiments (drones, legged robots, manipulators with tactile inputs) remains untested.
• Real-time and safety-critical deployment: The report lacks latency, compute, and closed-loop driving/robotics evaluations; moving from benchmark scores to safety-certified, real-time operation is an open engineering problem.
• Data curation and label quality: The training set is assembled from dozens of existing datasets with heterogeneous annotation quality; understanding how label noise propagates through stages 3 and 4 would guide future dataset design.

Target Audience

Robotics and autonomous driving researchers building embodied foundation models; VLM practitioners interested in cross-domain transfer and post-training recipes (SFT to CoT to RL); industry engineers evaluating open-source alternatives to proprietary multimodal models for physical-world deployment; and benchmark designers seeking insight into how embodied capabilities should be measured across indoor and outdoor settings.

Authors’ abstract

We open-source MiMo-Embodied, the first cross-embodied foundation model to successfully integrate and achieve state-of-the-art performance in both Autonomous Driving and Embodied AI. MiMo-Embodied sets new records across 17 embodied AI benchmarks in Task Planning, Affordance Prediction and Spatial Understanding, while also excelling in 12 autonomous driving benchmarks across Environmental Perception, Status Prediction, and Driving Planning. Across these tasks, MiMo-Embodied significantly outperforms existing open-source, closed-source, and specialized baselines. Our results indicate that through multi-stage learning, curated data construction, and CoT/RL fine-tuning, these two domains exhibit strong positive transfer and mutually reinforce one another. We provide a detailed analysis of our model design and training methodologies to facilitate further research. Code and models are available at https://github.com/XiaomiMiMo/MiMo-Embodied.