RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model

AI summary

Overview

Research area: Robotics / embodied AI — embodied foundation models and vision-language-action (VLA) policies.

Technical level: Advanced. The paper is a large-scale technical report covering pretraining infrastructure, data mixtures, and real-robot deployment across three model scales.

Scope (one sentence): The paper introduces RynnBrain 1.1, a family of embodied foundation models at 2B, 9B, and 122B-A10B scales that adds contact-point prediction and native 3D grounding to RynnBrain 1.0, plus a post-trained RynnBrain-VLA with a unified cross-embodiment action space evaluated on three real robot platforms.

What This Paper Is About

RynnBrain 1.0 showed that a single multimodal model could handle egocentric understanding, spatial grounding, physically grounded reasoning, and planning, but left two questions open: how to bring its representations and outputs closer to actual robot manipulation, and how well it could serve as an initialization for downstream VLA post-training. RynnBrain 1.1 addresses both by adding contact-point prediction and native 3D grounding to embodied pretraining, and by building RynnBrain-VLA, a policy that can be trained jointly across robots with incompatible low-level action spaces.

Key Contributions

1. A three-scale embodied foundation model family. RynnBrain 1.1 is released at 2B, 9B, and 122B-A10B parameter scales, all built on Qwen3.5 variants and trained under one capability definition and training framework. The paper reports an upward capability trend from 2B to 9B to 122B-A10B, and states that RynnBrain 1.1-122B-A10B surpasses all proprietary and open-source models evaluated on VSI-Bench, MMSI, and RefSpatial-Bench.

2. Two new pretraining tasks aimed at manipulation. Contact point prediction (across the model family) and native 3D grounding (for the 2B and 9B models) are added to embodied pretraining, described as producing representations and outputs more directly aligned with robot manipulation.

3. RynnBrain-VLA with a unified cross-embodiment action space. A single VLA recipe combines a shared 81-dimensional action space, partitioned into semantically aligned body-part groups, with embodiment-specific masking so that only the dimensions available on each robot receive loss.

4. Real-robot validation across heterogeneous platforms. RynnBrain-VLA is deployed on a Unitree G1 humanoid, an Astribot-S1 bimanual robot, and a Tianji-Wuji dexterous-hand system. Controlled evaluations show RynnBrain-initialized policies outperform Qwen-based VLA policies and representative generalist VLAs, and that joint multi-task and multi-embodiment training improves average process score and final success rate over per-task policies.

Main Findings

• Scale trend in embodied cognition and localization. The paper reports an upward trend in base-model performance from 2B to 9B to 122B-A10B across spatio-temporal understanding, spatial reasoning, localization, and 3D grounding benchmarks. On the 2B row reported in Table 3, RynnBrain 1.1-2B reaches 72.9 on VSI-Bench and 40.5 on MMSI, versus 70.5 and 34.1 for RynnBrain 1.0. On the 9B row in Table 4, RynnBrain 1.1-9B reaches 74.9 on VSI-Bench, 47.0 on MMSI, 86.9 on MindCube, and 67.2 on RefSpatial-Bench.

• 122B-A10B leads on three benchmarks. The abstract and contributions state that RynnBrain 1.1-122B-A10B outperforms all evaluated proprietary and open-source models on VSI-Bench, MMSI, and RefSpatial-Bench. The per-benchmark numeric results for the largest scale are not included in the provided excerpt, which is truncated after Table 4.

• Gains on localization tasks versus RynnBrain 1.0. At 2B, RynnBrain-Grounding rises from 79.1 (RynnBrain 1.0) to 84.1, RynnBrain-Area from 54.6 to 57.2, RynnBrain-Affordance from 89.4 to 90.2, and RynnBrain-Trajectory from 66.6 to 68.8. At 9B, RynnBrain-Grounding rises from 81.6 to 85.4, RynnBrain-Area from 56.2 to 59.6, and RynnBrain-Trajectory from 64.5 to 70.0.

• Some spatial benchmarks regress relative to RynnBrain 1.0. Reported values show RynnBrain 1.1-2B below RynnBrain 1.0 on RoboSpatial-Pointing (59.6 vs 65.6), RoboSpatial-VQA (60.4 vs 64.9), RoboSpatial-Overall (62.1 vs 65.2), and EmbSpatial (73.9 vs 79.6). At 9B, RoboSpatial-VQA is 68.0 versus 80.0 and RoboSpatial-Overall is 69.1 versus 73.1, while EmbSpatial improves to 81.9 from 80.4.

• Contact point representation replaces grasp rectangles. RynnBrain 1.0 used the four corners of an oriented rectangle for planar grasps; RynnBrain 1.1 adopts a compact contact-centered representation a = (p, θ), where p = (x, y) is the annotated grasp center and θ is the in-plane angle of the line connecting the gripper fingers, normalized to [0, 1000] and serialized as <grasp pose> (x, y), θ </grasp pose>. The paper argues IoU against a single reference rectangle may not correlate with execution success because grasp extents depend on gripper geometry and annotation conventions.

• A 2.6M-sample contact-point corpus. After view-conditioned rendering, projection, and filtering of simulation-native annotations, the corpus contains 2.6M training samples drawn from Grasp-Anything, Jacquard V2, GraspFactory, GraspNet-1B, and GraspClutter6D. The pipeline discards views where the object is excessively near, distant, truncated, or visually ambiguous, or where the grasp configuration is not observable.

• Native 3D grounding as a 9-dimensional prediction. For the 2B and 9B models, the model takes a language instruction and camera intrinsics and predicts a 3D bounding box in the camera coordinate system: center (cx, cy, cz), dimensions (w, l, h), and orientation (pitch, yaw, roll) in meters and radians, all discretized into integer tokens. This format follows Seed-VL and Qwen3-VL. The 3D-grounded dataset combines WildDet3D Essential (102,979 images, 374K manually verified annotations, 12,064 categories), WildDet3D Synthetic (896,004 images, 888K automatically selected annotations, 11,896 categories), and FoundationPose (1,849K images, 928 instances).

• Unified cross-embodiment action space. Actions are organized into an 81-dimensional shared space with groups Arm-Joint (14D, up to 7D per arm), Arm-EEF (18D, up to 9D per arm), Gripper (2D), Hand (40D), Torso (4D), and Head (3D). Unitree G1 activates 14D of the Hand group plus a separately predicted 64D SONIC latent token, concatenated into a 78-dimensional representation for the SONIC whole-body controller; Tianji-Wuji activates Arm-Joint (14D) and Hand (40D) for 54 active dimensions; Astribot S1 activates Arm-Joint (14D), Gripper (2D), Head (3D), and Torso (4D) while masking dexterous-hand joints.

• Deployment and control details. RynnBrain-VLA uses a flow-matching framework in which the VLM backbone acts as a single-stream Diffusion Transformer, predicting a 32-step action chunk. The control layer runs a low-frequency loop at 30 Hz and a high-frequency loop at 200 Hz. Real-Time Chunking triggers new inference every 5 steps with guidance strength β = 10.0.

• Controlled fine-tuning setup. The Qwen-based and RynnBrain-based VLA policies use identical demonstration data, pipeline, and optimization settings: 60k fine-tuning steps, learning rate 2 × 10⁻⁵, batch size 32, and images resized so the short side is 384 pixels.

• Joint multi-task, multi-embodiment training helps. The paper reports that joint multi-task and multi-embodiment training improves average process score and final success rate over separately fine-tuned per-task policies in controlled evaluations on Astribot and Tianji-Wuji.

• Not reported in the provided content. The specific numerical results for native 3D grounding benchmarks, contact-point prediction metrics, and the full real-robot success-rate tables are not included in the excerpt, which is truncated mid-way through Table 4.

Methodology in Plain English

The researchers start from an existing recipe—RynnBrain 1.0—and extend it in two directions rather than redesigning it. First, on the model side, they keep a decoder-only vision-language architecture with a vision encoder, a projector, and an LLM backbone initialized from Qwen3.5 at three sizes (2B, 9B, 122B-A10B), and reuse the same training pipeline across all sizes so that performance differences reflect scale and data rather than infrastructure changes. They also adopt DeepStack and Interleaved MRoPE to integrate multimodal information and handle long-context spatio-temporal modeling.

Second, on the data side, they keep the two organizing principles from RynnBrain 1.0—spatio-temporal memory and physical-world grounding—and add two new supervision types. Contact point prediction asks the model to output the grasp center and gripper rotation angle instead of a full grasp rectangle, which the authors argue is a better match to how manipulation actually works. Native 3D grounding asks the model to emit a 3D bounding box in camera coordinates, which the authors argue is directly usable by robots operating in three-dimensional space.

Everything is trained with a single autoregressive objective: the model minimizes negative log-likelihood over sequences that mix language tokens with spatial tokens for boxes, points, trajectories, and contact signals, so there are no separate task-specific prediction heads. Both language and continuous spatial coordinates are produced through the same prediction interface, with image coordinates mapped to integers in [0, 1000] and 3D quantities discretized into integer tokens.

For the action side, they post-train a VLA in which the VLM backbone itself serves as a single-stream diffusion transformer under a flow-matching objective. Actions from different robots are written into one shared high-dimensional vector, split into semantically aligned body-part groups, and then masked so that each robot only receives gradient on the dimensions it physically has. This lets one policy be trained on mixed data from robots with incompatible low-level interfaces, and lets a new robot be added by implementing only an embodiment dispatch layer rather than retraining the model.

Evaluation proceeds in two parts: benchmarking the base models on embodied multimodal tasks against a wide field of baseline models, and then running controlled real-robot comparisons where only the base model differs.

Why This Matters

Impact on research. The paper tests the hypothesis that large-scale embodied pretraining transfers to real-robot control by holding the downstream VLA pipeline fixed and varying only the base model. It also introduces a concrete mechanism—a masked, unified action space—for training one policy across robots with incompatible action spaces without forcing explicit alignment between them. On the data side, it argues against the grasp-rectangle convention that has been standard in grasp detection and replaces it with a contact-centered (p, θ) representation, which is a methodological claim other embodied work can test directly.

Real-world applications (as described or implied in the paper):

• Humanoid whole-body control, demonstrated on the Unitree G1 through a SONIC whole-body controller that consumes a 78-dimensional representation.
• Bimanual manipulation with parallel grippers and torso/head positioning, demonstrated on the Astribot-S1.
• Dexterous-hand manipulation, demonstrated on the Tianji-Wuji system combining a Tianji arm and a Wuji hand.
• Instruction-following tabletop tasks such as pouring liquid between containers and placing objects, which appear as the example instruction format in the VLA section.

Industry relevance. The work comes from DAMO Academy, Alibaba Group and Hupan Lab, with hardware support from WUJI TECH, and is positioned as a foundation model family that downstream teams fine-tune rather than train from scratch. The three-scale release (2B, 9B, 122B-A10B) is explicitly framed as a way to study how embodied capabilities evolve with scale, which matters for teams choosing a compute budget. The paper also reports controlled comparisons against Qwen-based and generalist VLA baselines under identical data and optimization settings, which is the comparison practitioners need when deciding what to initialize from.

Future Directions

• Extending native 3D grounding beyond 2B and 9B. The paper describes 3D grounding as available for the 2B and 9B-sized models, leaving open whether the same explicit 3D supervision helps or is necessary at the 122B-A10B scale.

• Understanding the spatial-benchmark regressions. RynnBrain 1.1 is below RynnBrain 1.0 on several reported RoboSpatial and EmbSpatial numbers at 2B and on RoboSpatial-VQA and RoboSpatial-Overall at 9B. The paper does not explain this in the provided content, which raises the question of what trade-off the new contact-point and 3D supervision introduces.

• Extending the unified action space to more embodiments. The framework is designed so that adding a robot only requires implementing an embodiment layer, but it is only instantiated on three platforms (Unitree G1, Astribot-S1, Tianji-Wuji). How well the 81-dimensional space covers a wider range of end-effectors, control modes, and degrees of freedom remains open.

• Closing the gap between action-grounding interfaces and executable grasps. The authors explicitly treat (p, θ) as an action-grounding interface rather than a complete executable grasp pose, noting that depth, approach direction, gripper aperture, collision constraints, and robot kinematics are resolved downstream. How much of that burden could be absorbed by the foundation model is left open.

Target Audience

This paper is most useful to robotics and embodied-AI researchers working on foundation models and VLA policies, particularly those interested in cross-embodiment training and in how much a pretrained multimodal backbone helps downstream manipulation. It also serves engineers building robot policies on top of an existing vision-language model, since much of the paper is a practical account of data construction, action-space design, deployment architecture, and fine-tuning hyperparameters. Readers interested in 3D grounding or grasp representation design will find the contact-point and 9-dimensional 3D box sections relevant. It assumes familiarity with autoregressive multimodal training and diffusion/flow-matching action policies.

Authors’ abstract

We present RynnBrain 1.1, a family of embodied foundation models spanning 2B, 9B, and 122B-A10B scales. Trained with a unified spatio-temporal and physically grounded framework, RynnBrain 1.1 supports embodied perception, spatial reasoning, localization, and planning. Compared with RynnBrain 1.0, it further introduces contact-point prediction across the model family and native 3D grounding for the 2B and 9B models, yielding representations and outputs that are more directly aligned with robot manipulation. We also develop RynnBrain-VLA with a unified cross-embodiment action space and embodiment-specific masking, and deploy it on Unitree G1, Astribot-S1, and Tianji-Wuji. RynnBrain 1.1 achieves strong results on embodied cognition, localization, and 3D grounding, with the 122B-A10B model outperforming all evaluated proprietary and open-source models on VSI-Bench, MMSI, and RefSpatial-Bench. Real-robot experiments show that RynnBrain-initialized policies outperform Qwen-based and representative generalist VLAs, while joint multi-task and multi-embodiment training improves process scores and success rates over per-task training.