ST4VLA: Spatially Guided Training for Vision-Language-Action Models

AI summary

Overview

• Research area: Robotics; specifically vision-language-action (VLA) models for robot manipulation, sitting at the intersection of multimodal foundation models and embodied control.
• Technical level: Intermediate. The paper is readable with a basic grasp of vision-language models, diffusion-style action heads, and imitation learning, but it assumes familiarity with terms such as grounding, affordance, co-training, and gradient alignment.
• Scope: One sentence — the paper proposes ST4VLA, a two-stage training recipe that injects spatial grounding priors into a dual-system VLA model and reports simulation and real-robot results.

What This Paper Is About

Large vision-language models can understand images and text well, but they do not automatically know where and how to move a robot arm. The authors show that naively fine-tuning such a model into a VLA, or co-training it on spatial and action data, causes the model to lose its spatial perception while it learns motor patterns. ST4VLA's goal is to keep spatial understanding intact during policy learning by pre-training the model on point, box, and trajectory prediction tasks and then conditioning the action module on spatially prompted latent features.

Key Contributions

1. Diagnosis of spatial collapse. The authors observe that directly fine-tuning a VLM with an action expert leads to a collapse of spatial priors, and that naïve co-training with spatial data introduces gradient conflicts between spatial grounding and action objectives; simple spatial prompting mitigates both issues.
2. A spatially guided training framework. ST4VLA explicitly aligns action optimization with spatial grounding objectives through a two-stage recipe: spatial grounding pre-training followed by spatially guided action post-training.
3. Architectural connection between planner and controller. A lightweight querying transformer of 8.7 MB maps variable-length VLM tokens into a fixed set of learnable query tokens, with a gradient decay factor (reported as a factor of 0.5) that attenuates gradients flowing from the action expert back into the VLM.
4. Broad empirical evaluation. The paper reports leading results on SimplerEnv, LIBERO, a 200-task Isaac-Sim pick-and-place benchmark, and real-robot pick-and-place and long-horizon tasks, comparing against baselines including π0 and GR00T N1.5.

Main Findings

• SimplerEnv gains from spatially guided training: Performance increases from 66.1 to 84.6 on Google Robot and from 54.7 to 73.2 on WidowX, which the authors describe as new state-of-the-art results on SimplerEnv.
• Comparison to prior state of the art: ST4VLA attains a 5.9% gain in Google Robot Visual Matching, a 5.3% gain in Visual Aggregation, and a 9.8% gain on the WidowX benchmark.
• Comparison to a matched Vanilla VLA baseline: Against a Vanilla VLA built on Qwen2.5-VL-3B-Instruct with a DiT action expert, ST4VLA achieves a 14.6% increase in Google Robot Visual Matching, a 12.4% increase in Visual Aggregation, and a 17.0% improvement on WidowX.
• Per-task SimplerEnv numbers (Google Robot): Visual Matching tasks — Pick Coke Can 97.3, Move Near 98.0, Open/Close Drawer 65.3, Open Top Drawer and Place Apple 77.8, average 84.6. Visual Aggregation — 95.6, 74.5, 68.0, 65.3, average 75.9.
• Per-task SimplerEnv numbers (WidowX): Put Spoon on Towel 80.2, Put Carrot on Plate 79.2, Stack Green Block on Yellow Block 35.4, Put Eggplant in Yellow Basket 98.0, average 73.2.
• Spatial perception degrades under action-only training: In the preliminary study, Vanilla VLA's RefCOCO-g performance drops to near-random levels by 20k steps.
• Spatially guided training balances both objectives: It maintains about 70% of original RefCOCO-g performance while reaching 60% WidowX success in just 20k steps.
• Gradient alignment measured by PSS: Vanilla co-training yields a Projection-Space Similarity of only 0.25, while spatially guided training raises it to 0.42.
• Multimodal understanding is preserved and improved: In the training-strategy study, the "+Spatially Pretrained" setting reaches MME 1411, MMVet 23.3, TextVQA 28.6, OCR 86.2, POPE 13.0/13.4, RefCOCO-g IoU@0.5 of 71.2, Refit-testB Acc@0.5 of 74.3, Google Robot 84.6/75.9, and WidowX 73.2, versus Vanilla co-training's MME 1106, RefCOCO-g 47.1, Refit-testB 66.7, Google Robot 70.2/66.5, and WidowX 61.1.
• Spatial priors raise the performance ceiling, not just convergence speed: When all models are trained to 100k steps, the authors report that baselines saturate at a significantly lower plateau than ST4VLA.
• Large-scale simulated pick-and-place: ST4VLA outperforms both π0 and GR00T N1.5 across all four evaluation tracks (in-distribution, unseen objects, new backgrounds, unseen instructions) on a benchmark of 200 tasks; the abstract states the approach improves large-scale simulation tasks by over 6% on average.
• Real-world pick-and-place generalization: ST4VLA averages 65 across settings versus 48 for GR00T N1.5 and 31 for π0. In-distribution it reaches 92, versus 78 for GR00T N1.5 and 45 for π0.
• Long-horizon manipulation: The abstract reports 92% success on real-world long-horizon manipulation under distribution shifts, and the paper states ST4VLA consistently surpasses GR00T N1.5 and π0 under in-distribution, physical interference, and task replanning settings.
• LIBERO results: ST4VLA achieves 98.0 on LIBERO-Spatial, 99.0 on LIBERO-Object, 93.8 on LIBERO-Goal, and 92.6 on LIBERO-Long, averaging 95.9 versus 94.2 for π0, 94.3 for π0.5-KI, 93.9 for GR00T N1, and 91.6 for the Vanilla VLA.
• Pre-training data ablation: With no additional pre-training the model scores 69.0 on Refit-testB, 66.1/63.5 on Google Robot, and 54.9 on WidowX; with general grounding data it reaches 30.7 Where2place accuracy, 74.9 Refit-testB, 72.6/70.3 Google Robot, and 65.2 WidowX. The third row of that table (robotic grounding data) is cut off in the provided content, so its numbers are not reported here.

Methodology in Plain English

ST4VLA separates the job of deciding what and where to act from the job of deciding how to act.

• Two systems. A "System 2" VLM planner (built on Qwen2.5-VL) reads images and instructions and produces latent planning tokens. A "System 1" action expert — a compact diffusion transformer (DiT) with a DINOv2 visual encoder — turns those tokens into motor commands.
• Stage 1, spatial grounding pre-training. The VLM is trained to predict points, boxes, and trajectories, using a mix of web-scale grounding data (RefCOCO, LLaVA-OneVision) and robot-specific data (RoboRefIt, A0, and the authors' own ST4VLA Data). All robot data is reformatted into a unified question-answering structure consistent with web-scale pre-training.
• Stage 2, spatially guided action post-training. The model learns embodiment-specific control while continuing to see spatial grounding data. For action data, a short spatial prompt is appended to the task instruction — for example, "store all toys into the toy box" becomes "Identify all relevant toys and their spatial relationships to the container" — which activates the spatial capability learned in Stage 1.
• Connecting the two systems. An 8.7 MB querying transformer uses cross-attention so that a fixed set of learnable query tokens attends to k intermediate layers of the VLM (k = 1 attends only to the final layer). A gradient decay factor (reported as 0.5) limits how much the action expert's gradients disturb the VLM, preserving the planner's semantic reasoning.
• Measurement tool. To test whether the two objectives pull in the same direction, the authors define Projection-Space Similarity (PSS), which compares the gradient subspaces of a spatial grounding loss and an action loss on shared parameters. In practice they compute this on a single 2048 × 2048 q-projection weight matrix in the final layer of the Qwen language model, using probing batches of size 64.
• Evaluation. They run a preliminary co-optimization study, then benchmark on SimplerEnv (Google Robot and WidowX), LIBERO, a 200-task Isaac-Sim pick-and-place benchmark built with GenManip (over 3,000 objects and containers, with 5 extra randomized-layout trajectories per task and post-training on 244K simulated demonstrations), and real-world tasks on a Franka Research 3 robot using 1K pick-and-place trajectories covering 23 objects and 5 containers. Long-horizon tasks (desktop sorting, drawer organization, sandwich making) use 22 hours of teleoperated demonstrations.

Why This Matters

Impact on research. The paper frames spatial grounding not as a side task but as a substrate that must be protected during policy learning. Its PSS measurement offers a concrete way to diagnose when two training objectives conflict, and its results suggest that the "collapse" of perception in VLA fine-tuning is addressable with prompting and gradient control rather than architectural overhaul. It also positions spatial prompting as complementary to unified architectures such as Magma, which the authors note uses spatial pre-training but not spatial prompting.

Real-world applications:

• Household and warehouse pick-and-place, where instructions must be followed despite clutter, novel objects, and rephrased commands.
• Long-horizon multi-step tasks such as desktop sorting, drawer organization, and food preparation, where a planner must decompose goals and recover from interference.
• Deployment in changing environments where lighting, background, and object layout differ from the training setup.
• Grasping and placement tasks where object pose and orientation at execution time differ from those seen in demonstrations.

Industry relevance. Robot foundation models are typically pre-trained on large action corpora; this paper shows that a comparatively lightweight post-training stage, combined with spatially grounded pre-training, can outperform heavily pre-trained baselines such as π0 and GR00T N1.5. The released code, data, and models at the project page lower the barrier to reproducing and extending the recipe.

Future Directions

• Unifying spatial and action objectives more tightly. The paper reports PSS rising only from 0.25 to 0.42, leaving roughly half the subspace misaligned; whether further alignment is achievable through other training schemes is an open question.
• Extending beyond manipulation. The authors describe spatial priors as embodiment-agnostic across manipulator joints, end-effector trajectories, humanoid locomotion, and mobile navigation, but all reported experiments use fixed-base arms (Franka variants, WidowX, Google Robot).
• Scaling the real-world evaluation. The real-robot pick-and-place study uses 1K trajectories covering 23 objects and 5 containers; whether the reported generalization holds at larger object and container diversity is not established by the reported data.
• Understanding the gradient decay factor. The decay is reported as 0.5 without a shown sweep in the provided content, so how sensitive results are to this value, and to the number of attended VLM layers k, remains an open question.

Target Audience

Researchers and engineers working on robot learning and foundation models, particularly those building VLA systems, studying multimodal co-training dynamics, or looking for practical recipes to preserve perception during policy fine-tuning. It is also relevant to practitioners deploying manipulation policies in real environments where objects, layouts, instructions, and backgrounds differ from training data. Readers without prior exposure to vision-language models or diffusion-based action policies will need background reading first.

Authors’ abstract

Large vision-language models (VLMs) excel at multimodal understanding but fall short when extended to embodied tasks, where instructions must be transformed into low-level motor actions. We introduce ST4VLA, a dual-system Vision-Language-Action framework that leverages Spatial Guided Training to align action learning with spatial priors in VLMs. ST4VLA includes two stages: (i) spatial grounding pre-training, which equips the VLM with transferable priors via scalable point, box, and trajectory prediction from both web-scale and robot-specific data, and (ii) spatially guided action post-training, which encourages the model to produce richer spatial priors to guide action generation via spatial prompting. This design preserves spatial grounding during policy learning and promotes consistent optimization across spatial and action objectives. Empirically, ST4VLA achieves substantial improvements over vanilla VLA, with performance increasing from 66.1 -> 84.6 on Google Robot and from 54.7 -> 73.2 on WidowX Robot, establishing new state-of-the-art results on SimplerEnv. It also demonstrates stronger generalization to unseen objects and paraphrased instructions, as well as robustness to long-horizon perturbations in real-world settings. These results highlight scalable spatially guided training as a promising direction for robust, generalizable robot learning. Source code, data and models are released at https://internrobotics.github.io/internvla-m1.github.io/