TRACE: Textual Reasoning for Affordance Coordinate Extraction

AI summary

Overview

• Research area: Robotics, specifically vision-language models (VLMs) for robotic manipulation, spatial affordance prediction, and Chain-of-Thought style reasoning for spatial grounding.
• Technical level: Intermediate. The reader needs some familiarity with VLMs, fine-tuning (full fine-tuning and LoRA), vision encoders, and attention-map analysis, but the core idea is explained in plain language.
• Scope (one sentence): The paper introduces a textual "Chain of Reasoning" (CoR) methodology and a 200,000-sample dataset (TRACE) that teach a VLM to write out its spatial reasoning steps before predicting 2D manipulation keypoints, evaluated on the RoboRefIt, Where2Place (W2P), and Where2Place (h) benchmarks.

What This Paper Is About

Robots driven by VLMs can understand high-level instructions, but they struggle to convert an ambiguous phrase like "find the free space in front of the window on the left" into precise low-level spatial coordinates for manipulation. Existing fixes either treat the VLM as a black box or use "visual" Chain-of-Thought that generates or edits intermediate images, which is computationally expensive. TRACE instead teaches the model to reason in text, producing an explicit step-by-step rationale that justifies the coordinate it finally predicts.

Key Contributions

1. The TRACE methodology: a framework that inserts a textual Chain of Reasoning into the affordance-prediction process, so a VLM externalizes why it chose a spatial point before outputting that point.
2. The TRACE dataset: a 200,000-sample training set built by an autonomous pipeline. It combines 100,000 novel reasoning-augmented samples (with explicit CoR text, generated by Gemini 2.5 Flash from RoboRefIt and Where2Place image-instruction pairs, 50,000 from each) with 100,000 standard visual instruction-tuning samples from LVIS and VQA, matching the data scale of the original RoboPoint model.
3. State-of-the-art spatial affordance results: instruction-tuning a VLM on this data improves precision on spatial benchmarks, with the strongest variant (RoboPoint(FFT)+TRACE) reaching 48.1% on Where2Place (a 9.6% relative improvement over the original RoboPoint) and 55.0% on the harder Where2Place (h) subset.
4. Evidence that the reasoning data itself causes the gain: an ablation shows performance rises monotonically as more CoR data is used, and an attention-map analysis shows the model's visual focus shifts across reasoning steps.

Main Findings

• Primary benchmark result: RoboPoint(FFT)+TRACE reaches 48.1% ± 0.1 on Where2Place, versus 43.9% ± 0.6 for RoboPoint(FFT), a statistically significant 9.6% relative improvement.
• Hard subset result: On Where2Place (h), which contains 30 examples with relation types unseen during training, TRACE reaches 55.0% ± 3.5 versus 46.9% ± 4.2 for RoboPoint(FFT).
• RoboRefIt result: TRACE reaches 42.9% ± 0.8 versus 41.7% ± 0.6 for RoboPoint(FFT).
• Full result table (metric: percentage of predicted points falling within the ground-truth mask, averaged over three runs):

• Reasoning data scales performance: In a 7B LoRA ablation sweeping the TRACE data from 0% to 100%, RoboRefIt rises 7.5 points (40.6% to 48.1%), Where2Place rises 7.6 points (36.1% to 43.7%), and Where2Place (h) rises 10.5 points (30.7% to 41.2%), a 34.2% relative gain over its baseline. The paper reports this as a statistically significant positive linear trend across all three benchmarks.
• Attention shifts across reasoning steps: For the instruction "Find the free space in front of the window on the left," attention is weak and diffuse during step (1) "Identify Reference Object" and step (2) "Define Target Area"; a distinct high-attention region emerges over the target area at step (3) "Determine Goal's Subtype"; and at step (4) "Generate Output" there is almost no visual attention on the predicted points, suggesting the model relies on its completed text reasoning rather than concurrent visual evidence.
• Statistical significance is partial: A two-sample t-test gives p = 0.022 on Where2Place (significant at p < 0.05), but p = 0.36 on RoboRefIt and p = 0.27 on Where2Place (h), which are not significant at that level; the model still consistently outperformed the baseline across all categories.
• Specialist and generalist VLMs trail far behind: SpaceLLaVA, GPT-4o, and Gemini were evaluated zero-shot and scored well below both RoboPoint variants on every benchmark (highest non-RoboPoint score was 20.0% on RoboRefIt by SpaceLLaVA).

Methodology in Plain English

The researchers start from RoboPoint, an existing VLM pipeline that predicts affordance keypoints, and enhance its data generation.

1. Generate reasoning text. They take existing image-instruction pairs with known ground-truth action points (50,000 from RoboRefIt and 50,000 from Where2Place) and use the Gemini 2.5 Flash model (gemini-2.5-flash-lite-preview-06-17) through its API to write a step-by-step rationale for each one. The prompt breaks the task into four cognitive steps: identifying reference objects, determining the goal's subtype (for example, "Placement Affordance"), defining the target area, and explaining how the final points were produced. This took roughly 40 hours of API computation time.
2. Assemble a comparable-size dataset. Those 100,000 reasoning-augmented samples are mixed with 100,000 standard instruction-following samples from VQA and LVIS, giving 200,000 training samples in total, matching the scale of the original RoboPoint training data so the effect of the reasoning text is isolated.
3. Frame training as next-token prediction. Given an image and an instruction, the model learns to output the reasoning text (on TRACE data) and then the target 2D coordinates, normalized to the range [0, 1] as a list of (x, y) tuples.
4. Train two model variants. The main 13B model (Vicuna-v1.5-13B) is fully fine-tuned for one epoch; the smaller 7B variant (Vicuna-v1.5-7B) is adapted with LoRA (rank r = 128, scaling α = 256) for the ablation study and attention visualization. Both use the CLIP-ViT-Large-Patch14-336 vision encoder (penultimate-layer features), a 2-layer MLP projector with GELU activation, Flash Attention 2, the Vicuna v1 conversation template, the AdamW optimizer at a global learning rate of 2e-6, cosine annealing with 3% warmup, bf16 mixed precision, and gradient checkpointing.
5. Optimize the data pipeline. Lazy preprocessing, padding images to a square aspect ratio, batching samples grouped by modality length, and 12 dataloader workers keep the pipeline from bottlenecking.
6. Evaluate and inspect. They compare against RoboPoint, GPT-4o, SpaceLLaVA, and Gemini on RoboRefIt (250 images), Where2Place (100 images), and Where2Place (h) (30 examples), then visualize attention maps to see how reasoning unfolds internally.

Why This Matters

Impact on research. The work offers a lightweight alternative to visual Chain-of-Thought, which typically requires generating future images, editing the input image, or running multi-step visual search, all of which add computational overhead. TRACE argues that reasoning in language, a VLM's native modality, is enough to sharpen spatial grounding. It also supplies a reusable 200,000-sample dataset and an autonomous generation pipeline built on top of the RoboPoint pipeline, plus evidence (dose-response ablation and attention analysis) that reasoning text is causally useful rather than decorative.

Real-world applications (the paper does not list deployed systems; these are directions its capability speaks to):

• Warehouse and logistics robots that must identify a free patch of shelf or conveyor space from a relational instruction such as "place it to the right of the bin."
• Home and service robots that need to place objects in open areas described only relative to furniture or windows.
• Industrial bin picking and assembly, where the target is defined by clutter and spatial relations rather than a fixed coordinate.
• Assistive or collaborative robots that must act on ambiguous human phrasing and whose decisions need to be auditable.

Industry relevance. One author is affiliated with ABB Robotics (Seoul), and the paper's framing targets a known pain point for commercial VLM-driven manipulation: reliability of low-level grounding in unstructured environments. Since the method adds only text during training and inference, it avoids the cost of running image-generation or image-editing models in the control loop. The published code and dataset (https://github.com/jink-ucla/TRACE) lower the barrier for teams building on RoboPoint-style pipelines.

Future Directions

• Richer, less-structured reasoning: The paper states that its reasoning chains are synthetically generated and may not capture the full complexity of human thought, and it plans to explore more sophisticated and less-structured reasoning generation.
• Broader task coverage: Extending the CoR framework beyond single-point spatial grounding to multi-step manipulation and navigation.
• Controllability and confidence: The model currently has no mechanism to explicitly control the reasoning process or to report confidence estimates for its predictions; the authors flag this as a limitation and propose investigating it.
• Using attention for reliability: Turning insights from attention maps into concrete improvements in model reliability and trustworthiness for VLM-driven robots.

Target Audience

• Robotics and embodied-AI researchers working on VLM/VLA-based manipulation and spatial affordance prediction.
• Practitioners who need reliable language-to-coordinate grounding but want to avoid the compute cost of visual Chain-of-Thought pipelines.
• Researchers studying reasoning in multimodal models, including those interested in how Chain-of-Thought transfers from text to spatial tasks.
• Engineers and teams with an existing RoboPoint-style system who want a drop-in, data-driven way to improve precision and interpretability, since the method depends on fine-tuning data rather than architectural change.

Authors’ abstract

Vision-Language Models (VLMs) struggle to translate high-level instructions into the precise spatial affordances required for robotic manipulation. While visual Chain-of-Thought (CoT) methods exist, they are often computationally intensive. In this work, we introduce TRACE (Textual Reasoning for Affordance Coordinate Extraction), a novel methodology that integrates a textual Chain of Reasoning (CoR) into the affordance prediction process. We use this methodology to create the TRACE dataset, a large-scale collection created via an autonomous pipeline that pairs instructions with explicit textual rationales. By fine-tuning a VLM on this data, our model learns to externalize its spatial reasoning before acting. Our experiments show that our TRACE-tuned model achieves state-of-the-art performance, reaching 48.1% accuracy on the primary Where2Place (W2P) benchmark (a 9.6% relative improvement) and 55.0% on the more challenging W2P(h) subset. Crucially, an ablation study demonstrates that performance scales directly with the amount of reasoning data used, confirming the CoR's effectiveness. Furthermore, analysis of the model's attention maps reveals an interpretable reasoning process where focus shifts dynamically across reasoning steps. This work shows that training VLMs to generate a textual CoR is an effective and robust strategy for enhancing the precision, reliability, and interpretability of VLM-based robot control. Our dataset and code are available at https://github.com/jink-ucla/TRACE