Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics

AI summary

Overview

• Research area: Robotics, embodied AI, spatial reasoning with vision-language models (VLMs).
• Technical level: Intermediate. The paper assumes familiarity with model fine-tuning concepts (supervised fine-tuning, reinforcement fine-tuning, rewards, encoders/decoders), but the core idea of "spatial tracing" is explained from first principles.
• Scope: The paper introduces RoboTracer, a 3D-aware VLM that performs multi-step, metric-grounded reasoning to turn spatially constrained instructions into ordered 3D keypoint sequences ("spatial traces"), along with a training dataset (TraceSpatial) and benchmark (TraceSpatial-Bench) to support it.

What This Paper Is About

Robots increasingly receive instructions that are constrained by space and physical size, such as "Water flowers from left to right with watering can hovering 1–5 cm above each one." Executing such an instruction requires first figuring out which objects are involved and where they are in 3D (3D spatial referring), and then understanding their actual real-world sizes and distances in meters or centimeters (3D spatial measuring). The paper's goal is to build a vision-language model that performs this multi-step reasoning and outputs a "spatial trace" — an ordered sequence of 3D points that a robot can follow — rather than the 2D point sequences or dense action predictions that prior systems produce.

Key Contributions

1. RoboTracer, a 3D-aware VLM that accepts arbitrary geometric inputs (RGB, camera intrinsics, depth), uses a universal spatial encoder plus a scale decoder supervised by a regression loss to gain real-world scale awareness, and applies metric-sensitive rewards during reinforcement fine-tuning to generate spatial traces.
2. TraceSpatial, a dataset of 4.5M high-quality examples and 30M QA pairs spanning outdoor, indoor, and tabletop scenes, with 48.2% absolute-scale data (14x a prior dataset), step-wise reasoning annotations up to 9 steps, and object-centric and end-effector-centric traces across 3 single-arm and dual-arm robot configurations.
3. TraceSpatial-Bench, a benchmark of 100 real-world images with manually annotated object localization, movement, and placement tasks requiring metric-grounded reasoning in up to 8 steps, with start-point masks, end-point 3D bounding boxes, and precise geometry annotations.
4. Real-world and simulated validation, showing integration with motion planning and control policies on UR5 and G1 humanoid robots in cluttered scenes, and use of the model as a tool to generate action data for other policies.

Main Findings

• Spatial understanding and measuring: RoboTracer-8B-SFT, trained only on TraceSpatial, reaches 96.62 on CV-Bench 2D-Relation, 97.17 on 3D-Depth, and 93.50 on 3D-Distance, and 83.01 on MSMU Scale Estimation and 70.00 on Refer Estimation. The paper reports an average success rate of 85.7% across Tables 1 and 3, exceeding Gemini-2.5-Pro by 8.58% and base model NVILA-8B by 20.3%. Improvements were larger on 3D-related and measurement tasks than on 2D tasks (23.6% vs. 14.7%).
• Abstract-level headline numbers: The abstract reports an average success rate of 79.1% on spatial understanding, measuring, and referring benchmarks, exceeding Gemini-2.5-Pro by 11%, and a 36% accuracy margin over Gemini-2.5-Pro on TraceSpatial-Bench.
• Benchmark leadership on spatial tracing: On TraceSpatial-Bench, RoboTracer-RFT with RGB plus intrinsics and depth input achieves 45% overall success rate, versus 3% for Gemini-2.5-Pro, 6% for Qwen3-VL-8B, and 28% for a finetuned RoboRefer-2B with RGB-D input.
• RFT beats SFT on multi-step tracing: Adding metric-sensitive process rewards improves overall success rate by 4% over purely outcome-based rewards. RFT without the spatial encoder slightly underperforms RFT with it but still largely exceeds SFT, and remains robust when depth input is noisy.
• 2D spatial referring and visual trace: RoboTracer-8B-SFT leads on Where2Place (69.00), RoboSpatial (66.40), and both Location (55.00) and Placement (53.00) parts of RefSpatial-Bench, and posts the lowest distances on ShareRobot-Bench and VABench-V (e.g., Discrete Fréchet 0.1449 and RMSE 0.0966 on ShareRobot-Bench).
• Geometry precision helps: Using more precise geometric cues yields up to a 6% absolute gain, and the model can incorporate additional geometry at inference without retraining.
• Simulated manipulation generalization: Across 19 RoboTwin 2.0 hard tasks (12 seen, 7 unseen in TraceSpatial), RoboTracer-2B reaches 75.4% success on seen tasks, 44.4% on unseen tasks, and 64.0% overall, outperforming the best baseline by 32.8% on unseen tasks, with no task-specific training.
• Real-world long-horizon tasks: On picking the rightmost hamburger and placing it on a keyboard without collisions, RoboTracer reaches 60.00% success versus 40.00% for a finetuned MolmoAct, 0.00% for MolmoAct and RoboRefer. On flower watering with a 1–5 cm hover, it reaches 30.00% versus 10.00% for finetuned MolmoAct and 0.00% for the others.
• Data generation utility: Using RoboTracer-generated spatial traces as auxiliary supervision raised MolmoAct's success rate from 0% to 25%.
• General capability preserved: On MME_test the model scores 1751 versus 1547 for NVILA-2B, and it stays roughly on par on MMBench_dev (77.62 vs. 78.63), OK-VQA (65.22 vs. 64.9), and POPE (82.52 vs. 81.96).
• Ablations: All three data sources (2D, 3D, video) are needed for best performance; the regression loss for scale supervision beats next-token prediction and no supervision; and the decoupled (u, v, d) point formulation outperforms direct (x, y, z) modeling.

Methodology in Plain English

The authors formulate spatial tracing as predicting 6–12 ordered 3D keypoints from visual input, optional camera geometry, and a text instruction. Rather than predicting (x, y, z) world coordinates directly, each point is expressed as an image coordinate plus an absolute depth (u, v, d), which converts to 3D using camera intrinsics and can be reduced to 2D tasks by dropping depth.

The base model is NVILA (2B and 8B variants). Two additions are attached to it: a universal spatial encoder built on a feed-forward metric 3D geometry model that can flexibly accept geometric cues such as camera intrinsics, poses, and depth, and a scale decoder that maps a special token embedding into a numeric scale factor, trained with a regression loss on the logarithm of the predicted scale (using only the projector and scale decoder in the first SFT stage).

Training happens in three stages. First, metric alignment: the spatial encoder and scale decoder are aligned with the language model using geometric annotations, updating only the projector and scale decoder. Second, metric enhancement: the spatial encoder is frozen and everything else is fine-tuned on TraceSpatial plus general instruction-following data, mixing RGB-only and RGB-plus-geometry inputs. Third, reinforcement fine-tuning with GRPO on multi-step reasoning data, using outcome-based rewards (format, endpoint consistency, trajectory alignment) plus metric-sensitive process rewards that check the format and accuracy of intermediate perceptual steps, scaled by 0.25. Points are normalized to [0, 1] with depth scaled by the scene's maximum depth. For real robot deployment, the predicted traces are corrected to satisfy physical constraints and then fed to motion planning, avoidance, and joint-limit checks.

Dataset construction combines three pipelines: filtered 2D web images for basic spatial concepts, 3D scans for metric-accurate indoor reasoning and occupancy maps, and real and simulated manipulation videos for gripper-centric traces.

Why This Matters

Impact on research. The paper reframes a hard robotics problem as a surrogate prediction task — generating a 3D spatial trace — that keeps metric grounding and intermediate reasoning explicit, instead of collapsing everything into dense end-to-end actions. It also argues that process-level rewards over key perceptual steps matter more than outcome-only rewards for tasks needing multi-step 3D reasoning, and shows the chosen (u, v, d) representation enables reuse of 2D data for co-training.

Real-world applications.

• Household and service robots handling instructions with metric constraints, such as pouring, watering, or placing objects with specified clearances.
• Warehouse and logistics picking, where "the rightmost item" and collision-free placement routes must be resolved dynamically as the scene changes.
• Laboratory or kitchen automation, where tasks like precise ingredient placement depend on absolute distances rather than relative 2D coordinates.
• Robot data generation, using the model's traces as auxiliary supervision to improve other policies — the paper reports raising MolmoAct from 0% to 25% success after such fine-tuning.

Industry relevance. The work targets a gap between powerful general VLMs and practical robot control: current strong models still produce floating or colliding traces because their depth and metric understanding is weak. Because RoboTracer is embodiment-agnostic, integrates with arbitrary geometric inputs without retraining, and can be executed by both a UR5 arm and a G1 humanoid, it points to a reusable spatial reasoning layer that could sit above existing motion planners — and the paper notes integration with Code-as-Monitor enabling updates at 1.5 Hz.

Future Directions

• Extending reinforcement fine-tuning to the larger model. The paper performs RFT only on the 2B model due to computational limits, so whether the 8B variant would gain more from process rewards is untested here.
• Broadening robustness and input flexibility further. The model already tolerates noisy depth and additional geometry, but the reported geometry gains (up to 6%) suggest headroom if metric cues become more reliable in real deployments.
• Expanding dataset and benchmark coverage. The pipeline is described as scalable to RGB-only data in new domains and is demonstrated on re-annotated data, but the benchmark itself contains 100 real-world images, so generalization to wider scene and task distributions remains open.
• Closing the remaining success-rate gap on difficult tasks. Real-world success rates of 60.00% and 30.00% on the two reported constrained tasks, and 44.4% on unseen simulated tasks, indicate that long-horizon, metrically tight manipulation is still far from solved.

Target Audience

Robotics and embodied AI researchers, VLM practitioners interested in spatial and metric reasoning, and engineers building manipulation systems that must follow spatially or metrically constrained natural-language instructions. Also useful for dataset and benchmark builders, since a substantial part of the contribution is the TraceSpatial data pipeline and TraceSpatial-Bench evaluation protocol. Readers should be comfortable with fine-tuning terminology but do not need deep prior exposure to 3D geometry models.

Authors’ abstract

Spatial tracing, as a fundamental embodied interaction ability for robots, is inherently challenging as it requires multi-step metric-grounded reasoning compounded with complex spatial referring and real-world metric measurement. However, existing methods struggle with this compositional task. To this end, we propose RoboTracer, a 3D-aware VLM that first achieves both 3D spatial referring and measuring via a universal spatial encoder and a regression-supervised decoder to enhance scale awareness during supervised fine-tuning (SFT). Moreover, RoboTracer advances multi-step metric-grounded reasoning via reinforcement fine-tuning (RFT) with metric-sensitive process rewards, supervising key intermediate perceptual cues to accurately generate spatial traces. To support SFT and RFT training, we introduce TraceSpatial, a large-scale dataset of 30M QA pairs, spanning outdoor/indoor/tabletop scenes and supporting complex reasoning processes (up to 9 steps). We further present TraceSpatial-Bench, a challenging benchmark filling the gap to evaluate spatial tracing. Experimental results show that RoboTracer surpasses baselines in spatial understanding, measuring, and referring, with an average success rate of 79.1%, and also achieves SOTA performance on TraceSpatial-Bench by a large margin, exceeding Gemini-2.5-Pro by 36% accuracy. Notably, RoboTracer can be integrated with various control policies to execute long-horizon, dynamic tasks across diverse robots (UR5, G1 humanoid) in cluttered real-world scenes. Please see the project page at https://zhoues.github.io/RoboTracer.