SLA: Scaling Spatial-Language-Action Model via Continual Pre-training

AI summary

Overview

• Research area: Robotics foundation models — vision-language-action (VLA) policies, 3D spatial representation learning, and continual pre-training (cs.RO).
• Technical level: Advanced. It assumes familiarity with VLA architectures, flow-matching action heads, feature alignment/distillation, and multi-dataset robot training pipelines.
• Scope: The paper proposes SLA, a training recipe that gives a pretrained 2D VLA policy transferable 3D spatial understanding by aligning its internal visual features with a frozen 3D perception model during large-scale continual pre-training and again during downstream post-training, without changing the inference architecture.

What This Paper Is About

Most VLA robot policies inherit their visual representations from 2D image-text pretraining, so any 3D spatial ability is bolted on late, during task-specific fine-tuning, and depends on sensors that are noisy, heterogeneous, or simply absent from many large robot datasets. The authors argue that spatial understanding should instead be built into the foundation model itself, at scale, from broad robot experience. Their goal is to develop that spatial competence through continual pre-training on a unified corpus of public robot data while preserving the pretrained policy's weights, architecture, and generalization.

Key Contributions

1. SLA model: A VLA training scheme that implicitly aligns intermediate visual embeddings of a policy with spatial representations from a frozen 3D perception foundation model (OmniVGGT), added as an auxiliary loss on top of the standard action-prediction objective. It requires no architectural change and no 3D input at inference.
2. Spatial continual pre-training at scale: The alignment principle is extended beyond post-training to a curated, unified corpus of six public robot datasets totalling roughly 1.7M episodes and 14,330 hours, with states and actions harmonized into a shared end-effector representation across embodiments.
3. Adaptive layer selection and state dropout: Feature-dependent (token- and channel-wise) gates aggregate visual tokens from multiple VLA transformer layers for spatial supervision instead of a single hand-picked layer, and probabilistic state dropout pushes the policy toward visual cues rather than memorized trajectories.
4. Broad empirical validation: State-of-the-art or competitive results on LIBERO-Plus, RoboTwin 2.0 (Clean2Random), RoboCasa-Panda, and RoboDojo-Real, plus eight real-world deployment conditions and controlled data-scaling experiments.

Main Findings

• LIBERO-Plus: SLA reaches an 87.4% weighted average success rate over 10,030 rollouts, improving on π0.5 (84.4%) by 3.0 points and on Spatial Forcing (85.1%) by 2.3 points, and exceeding Qwen-RobotManip (87.1%) by 0.3 points. Gains over π0.5 appear in all seven perturbation dimensions, including observation noise (89.0% to 95.4%) and background (94.1% to 98.0%); its lighting result of 98.8% is the highest among compared methods.
• RoboTwin 2.0: With training only on clean demonstrations, SLA achieves 86.0% Clean2Clean, 72.3% Clean2Random, and a 79.2% average across the two settings — 26.3 and 20.9 points above π0.5 on Clean2Random and average respectively, and 10.5 points above post-training-only Spatial Forcing. It beats ME-Dex-1.0 (78.9% average) by 4.2 points on Clean2Random while trailing that method on Clean2Clean (89.6%).
• RoboCasa-Panda: 68.4% success using only 50 downstream demonstrations per task, which the authors report as exceeding compared policies trained with 300–3,000 demonstrations per task (3,600 evaluation trials total).
• RoboDojo-Real: A single policy adapted across ARX X5, Piper, and Piper X reaches 33.9% overall success versus 12.8% for π0.5.
• Real-world deployment: Across eight conditions (A–H) on a bimanual Piper platform and a single-arm Franka robot, SLA has the highest success rate in every condition, averaging 83.8% and improving on π0.5 by 24.0 percentage points. Conditions probe bimanual coordination, spatial and dynamic generalization, few-shot cross-embodiment adaptation, and instruction following.
• Data scaling: Controlled experiments show that applying spatial alignment in both continual pre-training and post-training outperforms either single-stage alternative at every tested data proportion.
• Diagnostic evidence: A depth-probing experiment (frozen VLA, only a DPT head trained) shows that embeddings learned from 2D images alone do not produce meaningful spatial structure, whereas aligned embeddings do.
• Not reported in the provided text: the per-task RoboCasa-Panda and RoboDojo-Real tables, the value of the alignment weight α, gate-behavior ablations, and the detailed results for deployment conditions A–H individually, since the paper content is truncated in Section 4.3.2.

Methodology in Plain English

The starting point is an off-the-shelf VLA policy (π0.5) that already knows how to turn images and instructions into actions. The authors leave its architecture untouched and instead add a training-time teacher: a frozen 3D perception model (OmniVGGT, built on VGGT) that looks at the same multi-view images and produces rich per-pixel spatial features. Available camera calibration and depth maps are fed to the teacher when present, and placeholder tokens cover the cases where they are missing, so datasets with partial or no geometry still contribute.

The policy's own visual tokens, taken from several intermediate transformer layers, are passed through layer normalization and a small shared MLP, weighted by learned per-token, per-channel gates, and summed. A cosine-similarity loss pulls this fused representation toward the teacher's resampled spatial features at the matching image location. Gradients flow into the policy, the projector, and the gates; the teacher stays frozen. The total objective is the usual action loss plus this alignment loss scaled by a weight α.

Because different data domains may favor different layers, the gates decide how much each layer contributes for each token rather than fixing one layer by hand. State dropout randomly removes the entire robot-state token segment per sample during training so the policy cannot lean on proprioception as a shortcut, while instructions and images are kept.

The data side is a substantial engineering effort. Six public sources (InternData-A1, AgiBotWorld-Beta, RoboCasa365-Pretrain, RoboCOIN, DROID, MolmoAct) are unified: all poses are expressed in a robot-centered end-effector frame (origin at the arm base, or the midpoint between arm bases for bimanual robots; +x forward, +y left, +z up), states are absolute end-effector poses, and actions are chunk-wise deltas computed with SO(3) composition and encoded as rotation vectors to avoid Euler-angle wrap-around outliers. Highly repetitive simulated episodes are downsampled (for example, 3% of episodes for selected InternData-A1 tasks and 10% for others; 0.5% of RoboCasa365 MimicGen episodes and 15% of composite episodes), and data using dexterous hands or robot-base motion is filtered out.

Continual pre-training runs 36k optimizer steps at an effective batch size of 4,608, with learning rate warmed to 5×10⁻⁵ over 1k steps and cosine-decayed to 1×10⁻⁵ at step 30k, taking about 110 hours on 48 NVIDIA H100 GPUs. Benchmark-specific post-training then reuses the same alignment principle (LIBERO-Plus: batch 512, 30k steps; RoboTwin: 256, 30k; RoboCasa-Panda: 64, 60k; RoboDojo-Real: 512, 40k; real-world: 64, 20k). At deployment, the teacher, projector, and gates are discarded, and the policy runs with its original inference graph and no depth or calibration inputs.

Why This Matters

• Research impact: The paper reframes 3D awareness in VLA models from a post-hoc add-on into a foundation-model capability acquired through continual pre-training, and it shows a way to do so without touching the deployed architecture or requiring 3D sensors at test time. The unified cross-embodiment action representation and the adaptive multi-layer gating are reusable ingredients for anyone training robot foundations on mixed public data.
• Real-world applications:
  • Warehouse and logistics picking, where lighting, background, and camera placement vary and policies must generalize beyond their demonstration setup.
  • Household service robots operating in cluttered kitchens, matching the RoboCasa-Panda setting.
  • Bimanual assembly and coordination tasks, matching the Piper deployments.
  • Low-data industrial cell deployment, where only tens to a hundred demonstrations per task are practical, as in the 50-demonstration RoboCasa-Panda and 100-demonstration real-world protocols.
• Industry relevance: Because SLA preserves the pretrained weights and adds zero inference overhead, it is a drop-in upgrade path for companies already building on open VLA checkpoints — the compute cost is concentrated in a one-time ~110-hour pre-training run rather than in per-task engineering or extra sensors.

Future Directions

• Beyond frozen-teacher alignment: The current teacher is fixed; studying joint or iteratively updated 3D supervision, and whether the alignment can be pushed into the earliest pretraining stages rather than continual pre-training on top of a 2D model, is a natural extension.
• Scaling laws for spatial supervision: The paper reports data-scaling experiments but the relationship between corpus size, alignment strength α, and spatial generalization is not fully characterized in the provided text.
• Dexterous manipulation and mobile bases: The corpus deliberately excludes dexterous-hand data and base motion, leaving open how the end-effector delta representation and spatial alignment extend to those settings.
• Longer-horizon and dynamic tasks: Real-world conditions include dynamic generalization, but extending spatially grounded policies to long-horizon, highly dynamic manipulation and to embodiments with very different kinematics remains untested.

Target Audience

Robotics and embodied-AI researchers and engineers working on VLA foundation models, cross-embodiment training, or 3D-aware manipulation policies. It is most useful to readers already comfortable with transformer-based policies and multi-dataset training pipelines who want a concrete, architecture-preserving recipe for injecting spatial supervision at scale; the benchmark suite and data-unification details also make it valuable for practitioners planning real robot deployments with limited demonstrations.

Authors’ abstract

Reliable robotic manipulation requires both semantic understanding and precise control in three-dimensional space. However, vision-language-action (VLA) foundations largely inherit visual representations from 2D vision-language pre-training, while spatial supervision is often introduced only during task-specific adaptation. Developing spatial understanding as a transferable foundation-model capability therefore calls for learning from broader robot experience. We present SLA, a Spatial-Language-Action Model that scales implicit spatial alignment to continual pre-training and maintains it during downstream post-training. Initialized from a pretrained pi05 policy, SLA jointly optimizes action prediction and alignment between intermediate visual embeddings and spatial representations from a frozen 3D perception model. We curate and unify heterogeneous robot datasets to support spatial learning across diverse embodiments, tasks, and scenes. To accommodate this diversity, we introduce feature-dependent gates that adaptively aggregate visual embeddings from multiple VLA layers for spatial supervision. The teacher further exploits partially available depth and camera calibration alongside RGB observations, enabling effective use of heterogeneous geometric annotations. Probabilistic state dropout encourages reliance on visual cues and mitigates overfitting to demonstration trajectories. All auxiliary components are used only during training, allowing direct reuse of pretrained VLA weights and preserving the original inference architecture.