ACE-Brain-0.6: Scaling Spatial Grounding for Physical Foundation Models

AI summary

Overview

Research area: Robotics and embodied AI, specifically visual/spatial grounding as the perceptual foundation for physical foundation models; the work sits at the intersection of vision-language model training, data engineering, and robot manipulation.

Technical level: Intermediate. The core ideas (grounding, question–answer supervision, data scaling) are explained accessibly, but familiarity with vision-language model fine-tuning and grounding benchmarks helps.

Scope: A data-centric study showing that scaling diverse, automatically annotated, verifiable grounding supervision lets an open 9B vision-language model match or exceed GPT-6 Astra on a range of public grounding benchmarks.

What This Paper Is About

General-purpose multimodal models such as GPT-6 Astra and Claude Opus 5.5 are becoming capable embodied operators, and the authors argue this ability rests on strong spatial perception: knowing where in a cluttered scene to act. Grounding errors can misdirect a robot's actions even when its high-level reasoning is correct, so the paper asks whether an open 9B model can close the gap to a frontier closed model purely by scaling the quantity and coverage of grounding training data, with the architecture and training recipe held fixed.

Key Contributions

1. A grounding data pipeline. The authors collect large-scale open-source robot, egocentric, object-centric, and spatial data, automatically annotate robot recordings (which rarely carry object labels), and convert every source into verifiable question–answer supervision covering target identification, complete instance coverage, and precise localization.
2. A controlled empirical study of grounding-data scaling. A series of training runs from the same base checkpoint with an identical recipe shows that grounding performance improves steadily with data scale, and that broadening the coverage of supervision matters far more than repeating existing sources.
3. ACE-Brain-0.6. Fine-tuning Qwen3.5-9B on roughly 1.5 million samples for a single epoch produces a model that beats GPT-6 Astra on four of seven public grounding benchmarks and has a higher seven-benchmark average (75.41 versus 73.68).
4. A robot-scene evaluation set. The paper introduces RobotGrounding, 2,591 questions in 14 types over held-out robot manipulation frames with human-annotated boxes, where ACE-Brain-0.6 scores 93.42 against Astra's 91.10.

Main Findings

• Beats the base model almost everywhere: ACE-Brain-0.6 improves on Qwen3.5-9B on six of seven public benchmarks and falls behind only on PixMo-Count (86.96 versus 90.35), where the base model was already strong. The seven-benchmark average rises by almost ten points, from 65.72 to 75.41.
• Ahead of GPT-6 Astra on aggregate: The model wins on four of seven benchmarks and on the average (75.41 versus 73.68), but trails Astra on PixMo-Count (86.96 vs 88.10), PointArena (74.13 vs 75.36), and clearly on RefSpatial (71.84 vs 80.14).
• Strongest open model of its size: It leads every compared open embodied model of the same scale — RynnBrain-8B, RoboBrain 2.5-8B, ACE-Brain-0.5-8B, Embodied-R1.5-8B, SenseNova-SI-1.1-8B, and PhysBrain 1.5-8B — on every benchmark except ShareRobot, and leads the next best on the seven-benchmark average by more than seven points.
• Best in robot scenes: On RobotGrounding, ACE-Brain-0.6 outperforms all compared models including the five closed ones. Most of its gain over the base model comes from three abilities the base largely lacks: recognizing a named object is absent, recognizing no object has a requested attribute, and matching an object across two camera views.
• Complementary strengths against Astra: On labeled subsets, ACE-Brain-0.6 leads on RoboSpatial compatibility (84.76 vs 76.2), RoboSpatial context (62.27 vs 53.4), PointArena reasoning (75.13 vs 56.3), all three RoboAfford subsets, and ShareRobot affordance; Astra leads on RefSpatial placement (84.0 vs 75.0), RefSpatial location (77.0 vs 73.0), and RoboSpatial configuration (86.2 vs 83.74).
• The two models fail differently: Astra often reports objects that are absent, whereas ACE-Brain-0.6 more often misses objects that are present. Astra stays ahead when every instance of a set must be enumerated or a target is defined by a spatial relation.
• Log-linear data scaling: Across data releases from 0.26M to 1.55M samples, the seven-benchmark average rises from 69.3 to 75.4 along a log-linear trend of about +2.4 points per doubling (r = 0.99), with no sign of saturation in the tested range.
• Coverage beats volume: A control run that doubles only the sources of an early release — reaching the size of the next release without adding any new task or domain — gains less than one point, landing within one point of the release that added new sources instead. Adding more samples whose answers list many points did not improve PointArena questions with many instances.
• Placement is not solved by more 2D data: Adding several thousand placement questions with targets computed from 2D object masks left the RefSpatial placement score unchanged, which the authors attribute to the need for 3D-verified targets.

Methodology in Plain English

Every grounding task is turned into text generation: given images and a natural-language query, the model emits one of five answer forms — a point, a list of points, a box, an integer count, or a categorical token (yes/no, or none for an absent target). Coordinates are integers normalized to [0, 1000].

Training data comes from three families. Robot manipulation recordings across more than thirty dataset families supply the target domain. Egocentric human–object interaction data adds first-person views with hand–object masks and contact regions. General pointing and grounding data contributes exhaustive instance sets for counting, part masks, 3D annotations, and a little general visual question answering to preserve broader ability.

Because robot recordings rarely have object labels, the pipeline annotates them automatically in three steps: pick a few keyframes per episode using the action signal (favoring gripper-state transitions and large action changes while keeping the trajectory's start, middle, and end); have a vision-language model name and describe the objects visible per episode and camera, forming an open vocabulary; then run Grounding DINO for detection with SAM 3 refining boxes, and use a SAM 3 recognition branch to relabel boxes while preserving their number and geometry.

Questions are generated from templates under rules that keep every answer checkable against its annotation: counting questions are emitted only when the instance set is complete, negative questions are verified against the image's vocabulary, and shortcuts are removed — no answer form exceeds 70% within a template, yes/no answers are balanced, and the number of requested points matches the answer. Detector boxes are VLM-verified, and all evaluation benchmark images and episodes are excluded from training. The result is about 1.54 million samples, roughly 3.41 million question–answer pairs over about 1.39 million distinct image paths.

Training fine-tunes all language-model parameters on Qwen3.5-9B with the vision encoder frozen and the multi-token-prediction head disabled, for one epoch, with loss only on answer tokens. Sequences are packed to 16,384 tokens with variable-length attention so different conversations never attend to each other; each step holds 16 such sequences, roughly 140 samples. The optimizer is AdamW with a learning rate warmed up over 50 steps to 1×10⁻⁵ then cosine-decayed to 1×10⁻⁷, chosen once by a four-point sweep on an early release and held fixed so that later differences reflect data alone.

Why This Matters

The paper argues that spatial grounding is the bridge between a model's reasoning and the physical world, and it demonstrates that this bridge can be substantially strengthened by better data rather than a bigger model. If a 9B open model can match a frontier closed model on grounding simply by scaling verifiable supervision, that changes where effort should go in building embodied systems.

Real-world applications:

• Robot manipulation: reliable picking, placement, and folding depend on localizing objects, parts, and free space precisely; the paper's own case studies involve basket placement on a support surface and folding a T-shirt using its measured extent.
• Warehouse and logistics robotics: counting and enumerating every instance of a set, and rejecting targets that are not actually present, are core skills the model targets.
• Assistive and service robots: first-person egocentric data and interaction-region grounding support hand–object contact tasks.
• Cross-view perception on multi-camera platforms: matching an object seen in one camera to the same object in a simultaneous view is a benchmarked capability.

Industry relevance: the work offers a reproducible recipe — annotation pipeline plus fixed fine-tuning — for turning existing open robot datasets into usable grounding supervision, which lowers the cost of building physical foundation models and gives a concrete, measurable target for data teams.

Future Directions

• Complete multi-instance supervision. Enumerating every instance of a set remains a weakness against Astra, which the authors attribute to a gap in the data rather than the model.
• Feasibility-aware placement data. Placement under spatial constraints requires targets verified in 3D, since the current free-space targets mark only unoccupied 2D regions; the authors state these are the next additions to the pipeline.
• Whether the log-linear trend continues. The 1.55M release shows no saturation, leaving open how far the trend extends beyond the tested range and whether growth eventually requires architectural change.
• Separating scale from coverage more cleanly. The main series grows sample count and supervision coverage together, so only a single control run isolates the two; further controls would sharpen the claim that coverage is what drives gains.

Target Audience

Researchers and engineers working on embodied AI, robot learning, and vision-language models — particularly those who build training data pipelines or evaluate spatial grounding. It is also useful for practitioners deciding whether to invest in model scale or data curation, and for readers wanting a concrete, quantitative comparison of an open 9B model against frontier closed models on grounding tasks.

Authors’ abstract

General-purpose multimodal models, including GPT-6 Astra and Claude Opus 5.5, demonstrate broad capabilities in embodied manipulation. Our analysis indicates that this generality is accompanied by strong spatial perception, as Astra can determine where to interact with task-relevant entities amid visual distractors. Because spatial perception connects high-level reasoning to the physical scene, we investigate scaling visual grounding to strengthen the perceptual foundation of general embodied operation. We develop a unified data pipeline that combines robot manipulation recordings with complementary visual data from diverse sources. After automatically annotating the robot recordings, the pipeline converts the collected data into verifiable grounding supervision. Fine-tuning Qwen3.5-9B on approximately 1.5 million samples produced by this pipeline yields ACE-Brain-0.6. The model outperforms GPT-6 Astra on four of seven public grounding benchmarks, with an overall average score of 75.41 compared with Astra's 73.68. Under a fixed Qwen3.5-9B fine-tuning setup, aggregate grounding performance improves steadily as successive data releases expand the training collection to cover a broader range of grounding scenarios. These results show that scaling verifiable grounding supervision from diverse sources can strengthen the spatial perception of an open 9B model for general embodied operation.