Grounded Action Model: 3D Grounding as a Foundation for Robotics

AI summary

Overview

• Research area: Robotics — robot foundation models, visuomotor manipulation policy learning, and 3D object grounding.
• Technical level: Advanced. The paper assumes familiarity with vision-language-action models (VLAs), world-action models (WAMs), diffusion/flow-matching action heads, and transformer conditioning schemes such as MM-DiT and adaptive layer normalization.
• Scope: The paper proposes and evaluates Grounded Action Model (GAM), a robot foundation model paradigm in which a frozen, promptable 3D grounding backbone replaces language or video generation as the pretrained foundation for manipulation, and evaluates it on two simulation benchmarks (RoboTwin 2.0, LIBERO-PRO) and two real robots (a bimanual YAM and a Franka).

What This Paper Is About

Manipulation policies need to know which objects matter and where they are in metric space, but current robot foundation models build on backbones pretrained for language generation (VLAs) or video generation (WAMs), neither of which directly requires explicit 3D grounding — so that grounding is learned implicitly from robot demonstrations and can fail when objects move, backgrounds change, or a different target is selected. GAM instead builds its policy on a pretrained promptable 3D grounding model (WildDet3D), which converts language, 2D point, or 2D box prompts into a shared object-centric representation that exposes which objects matter, where they are, and what geometry they occupy. Robot demonstrations are then used only to teach the policy how to act on those already-grounded objects.

Key Contributions

1. A new foundation paradigm. The paper introduces Grounded Action Models (GAMs), robot foundation models built on a pretrained promptable 3D grounding backbone rather than on a language-generation or video-generation backbone. The grounding backbone stays frozen during policy training, separating the acquisition of grounding capability from learning to act on grounded objects.
2. A unified prompt interface for task specification. Language, 2D points, and 2D boxes are all mapped into one object-centric representation, so a VLM planner (Molmo2) can specify targets directly through points while language specifies the operation to perform. This enables composition with semantic reasoning and episode memory without changing the action policy across sub-tasks.
3. An object-centric observation and action head. GAM builds image tokens (restricted to task objects and the robot arm) and detection tokens (encoding each object's point cloud, center, extents, and rotation), combines them with a state-history token, and predicts chunks of absolute joint-position targets with a 12-block MM-DiT action head trained by flow matching.
4. Broad empirical validation plus ablations. The paper reports results on RoboTwin 2.0 (50 tasks), LIBERO-PRO (16 perturbation settings), and two real robots, alongside ablations isolating the contribution of object-centric filtering and of each observation stream.

Main Findings

• RoboTwin 2.0 average success: GAM achieves 55.3% (55.29% in Table I) average success across the 50 RoboTwin 2.0 tasks, versus 52.0% for Spatial Forcing. GAM's action policy is trained only on the benchmark's 50 clean-scene demonstrations per task, under the official single-task protocol.
• Randomized-scene robustness: Under scene randomization (Hard), GAM reaches 47.6% (47.62% in Table I) versus 30.4% for the next best entry (Abot-M0), and retains 76% of its clean performance (63.0% to 47.6%). The strongest co-trained VLA and WAM entries reach higher clean success (77.2% for Spatial Forcing, 77.8% for FastWAM) but fall to 26.7% and 1.9% under randomization.
• 3D input alone is not enough: The 3D policy DP3 consumes the scene point cloud directly and reaches 55.24% clean, yet drops to 4.96% under randomization — showing that metric input without grounded selection does not confer robustness.
• Same-protocol comparison: Against π0.5 trained and evaluated under the same single-task protocol, GAM outperforms by 10 points (55.3% vs. 45.0%).
• LIBERO-PRO state of the art: GAM achieves the highest average across 16 perturbation settings at 0.61, versus 0.53 for π0.5. The largest gains are under Pos (target relocated) and Task (newly designated target), where GAM reaches 0.60 on Spatial-Pos, 0.47 on Object-Pos, 0.88 on Spatial-Task, and 0.50 on Object-Task. Every baseline stays below 0.11 on all four suites under Task.
• A known weakness: GAM trails under the Obj perturbation (object appearance and size) on all four suites, most notably on LIBERO-Goal (0.69) and LIBERO-10 (0.50), and π0.5 leads on Goal-Pos (0.38 vs. 0.19). The paper attributes this to difficulty adjusting grasping motions to object sizes not encountered in training.
• Bimanual YAM real-robot results: Both GAM and π0.5 achieve 19/20 in-distribution successes. Under visual shift, GAM retains 17/20 successes while π0.5 falls to 4/20 — retention rates of 89% and 21%, respectively.
• Franka planner composition: GAM composed with a Molmo2 planner achieves 64.7% in-distribution step completion, versus 42.4% for π0.5 and 35.3% for MolmoAct2. Out of distribution, it achieves 49.8%, versus 24.0% for π0.5 and 17.1% for MolmoAct2 — retaining 77% of in-distribution performance versus 57% and 48%.
• Ablation — object-centric filtering matters: On 10 RoboTwin 2.0 tasks (20 rollouts per task in Easy and Hard), full GAM averages 46.8%. Removing image masking drops it to 22.3%, removing point cropping to 12.8%, and removing both to 10.3%. On Easy the drop is 51.5% to 17.0%; on Hard, 42.0% to 3.5%.
• Ablation — both streams are needed: Detection-only tokens yield 16.0% average success and image-only tokens yield 20.3%, versus 46.8% for the full model. Neither stream alone matches their combination on Easy or Hard.
• Attention behavior: Qualitative attention maps on "Clear the table" show GAM's action-head attention remaining concentrated around selected targets, while baseline attention is more dispersed over perturbed backgrounds and distractors.

Methodology in Plain English

GAM splits the problem into a perception half that is pretrained and frozen, and an action half that is trained from robot demonstrations.

Perception (frozen). Given an RGB image and a task specification — language, 2D points, or 2D boxes — the system identifies which objects the task involves. For language, a span-tagging head trained on synthesized instruction templates over a frozen Flan-T5 encoder extracts the task-relevant object phrases; for example, "pick up the sponge and place it in the bowl" yields sponge and bowl, used as separate grounding queries. Those queries go to WildDet3D, which returns a 2D box and a metric 3D box per object, plus a dense metric depth map and the dense features of its visual backbone.

Turning detections into an observation. The backbone features are pooled onto a 16×16 grid, and only cells overlapping either a task object's 2D box (covering at least 30% of the cell) or the robot-arm silhouette (covering more than 10%) are kept; all other cells are zeroed out. The arm silhouette comes from posing the robot's URDF meshes via forward kinematics and projecting them with the known camera calibration. For each object, the depth map is back-projected into the robot base frame to form a scene point cloud; points inside a slightly inflated 3D box are cropped out, and 512 are sampled as the object's point cloud. A shared per-point MLP plus pooling (global max, global mean, and max-pooling within each of eight octants around the centroid) produces a 512-dimensional shape feature. The object descriptor bundles the normalized 2D box, the center (plus Fourier features at eight frequencies), the shape feature, the extents, and a 6D rotation, and a linear adapter turns it into 8 detection tokens. Undetected slots use a learned null token so the token budget stays fixed.

Action prediction (trained). Image tokens, detection tokens, and a state-history token built from the current and previous joint positions are given modality embeddings, passed through a shared self-attention layer for fusion, and fed as separate streams into a 12-block MM-DiT action head. Each block keeps separate parameters per stream, with timestep and language conditioning injected via adaptive layer normalization (the language projection is zero-initialized so the model starts unchanged). The head predicts a chunk of absolute joint-position targets plus gripper commands using flow matching, and at inference the chunk is produced by integrating the predicted velocity over K Euler steps from Gaussian noise. Only the action head — adapters, point-cloud encoder, fusion layer, language projection, and action transformer — is optimized during policy training.

Deployment. CoTracker3 tracks points inside the initial detection boxes and feeds them back as prompts, preserving object associations across frames. For long-horizon and memory-dependent tasks, a high-level planner (Molmo2) picks new targets at sub-task transitions using the current image and episode memory, reinitializing trackers through new point prompts, while the language instruction specifies the operation and the action policy itself never changes.

Benchmark adaptation. Because the grounding backbone is pretrained on real images, it is first fine-tuned on the rendered RoboTwin 2.0 domain using 2D boxes, 3D boxes, and depth exported directly from simulator state — no manual annotation — and then frozen before per-task action-head training.

Why This Matters

Impact on research. The paper reframes a central question — what should a robot foundation model learn during pretraining? — and offers a concrete alternative to language and video generation. It shows that object-centric observation design, rather than a stronger perception backbone, drives much of the robustness gain (all ablation variants share the same frozen backbone), and it introduces a prompt interface that lets a separate VLM planner drive a low-level controller through points rather than language. The reported gap on the Obj axis also marks a clear boundary of the approach.

Real-world applications (implied by the evaluated tasks):

• Bimanual pick-and-place and container placement in cluttered, visually varied settings, as tested on the bimanual YAM.
• Multi-step sorting and table clearing, as tested in the long-horizon Franka tasks ("Sort objects into three bowls", "Clear the table").
• Memory-dependent manipulation where the target is no longer visible in the current image, as tested in "Uncover the R, G, B blocks in order" and "Restore original layout".
• Human-in-the-loop target selection on real hardware, where a person clicks or draws a box to designate which object to manipulate.

Industry relevance. The work suggests that grounding backbones could be reused across embodiments — the same frozen grounding model is paired with different action heads, robots, and planners — and that robustness to scene randomization can be improved without training on randomized data, which matters for deployments where collecting demonstrations under every lighting, layout, and distractor condition is impractical. The point interface also gives integrators a clean seam between a high-level reasoning model and a low-level controller.

Future Directions

• Grounding quality as the bottleneck. The paper states that grounding errors propagate to actions without recovery, so more diverse 3D grounding datasets and grounding models that generalize across objects and scenes are needed.
• Richer geometric representations. The 3D bounding boxes used here are described as one possible form of grounding; the authors point to alternatives such as dense 3D instance segmentation for finer-grained geometric supervision.
• Grounding-aware failure recovery. Recovering from grounding mistakes is named as an open direction, since the current pipeline has no mechanism to detect or correct them.
• Adaptive context selection. Object-centric filtering can omit relevant context such as unselected obstacles; the authors plan to explore adaptive context selection for safe execution.

Target Audience

This paper is most valuable to robotics researchers and engineers working on manipulation policies, robot foundation models, and visuomotor generalization — particularly those evaluating VLAs and world-action models on RoboTwin 2.0, LIBERO-PRO, or real hardware. It is also relevant to researchers interested in 3D grounding and open-vocabulary detection, to teams building systems that compose a high-level VLM planner with a low-level controller, and to practitioners who need policies that stay reliable when scenes are randomized or targets are re-specified. Readers without a background in transformer architectures, flow matching, or action-chunking policies will find the methodology section demanding; the results and ablations remain interpretable without that background.

Authors’ abstract

Manipulation policies must know which objects matter and where they are, yet the pretrained backbones that current robot foundation models build on, from language in vision-language-action models (VLAs) to video generation in world-action models (WAMs), do not directly require this metric grounding, leaving it to be learned implicitly from robot demonstrations. We propose Grounded Action Models (GAMs), a new paradigm of robot foundation models built with 3D grounding. GAM can be conditioned using language, points, or box prompts, which are first transformed into a shared object-centric representation of the selected objects. This representation captures target-focused visual features and metric object geometry, which is mixed with robot state history through a multi-stream transformer to predict action chunks. Although GAMs can be run autonomously, they can also serve as a low-level controller that a high-level planner controls using its various input modalities, allowing for long-horizon and memory-dependent manipulation. On RoboTwin 2.0, GAM achieves an average success rate of 55.3% across 50 tasks (vs. 52.0% for Spatial Forcing), including 47.6% under scene randomization (vs. 30.4% for Abot-M0), with its action policy trained only on clean-scene demonstrations. On LIBERO-PRO, it achieves a state-of-the-art average success rate of 61% (vs. 53% for $π_{0.5}$) across 16 perturbation settings, with the largest gains when targets are relocated or newly designated. On two real robots, GAM retains 17/20 successes under visual shift on a bimanual YAM versus 4/20 for $π_{0.5}$, while its composition with a Molmo2 planner on a Franka achieves 64.7% ID and 49.8% OOD step completion on long-horizon and memory-dependent tasks.