Prompt-Guided Spatial Understanding with RGB-D Transformers for Fine-Grained Object Relation Reasoning

AI summary

Overview

• Research area: Computer vision and vision-language models (VLMs) applied to spatial reasoning in 3D industrial environments, specifically warehouse-scale scenes.
• Technical level: Advanced (assumes familiarity with vision-language architectures, depth estimation, fine-tuning, and benchmark evaluation protocols).
• Scope: The paper describes a depth-enhanced, prompt-augmented vision-language pipeline built on SpatialBot, fine-tuned on the Physical AI Spatial Intelligence Warehouse dataset for four spatial question categories, and reports its placement in Track 3 of the 2025 AI City Challenge (arXiv:2510.11996v1, 13 Oct 2025).

What This Paper Is About

Spatial reasoning in cluttered 3D environments such as warehouses is difficult for vision-language systems because scenes contain occlusion, clutter, and many similar objects, and most existing models rely on local appearance rather than explicit geometric grounding. The authors' goal is to improve a VLM's ability to answer fine-grained spatial questions about object layout, distance, count, and left-right relations by injecting explicit bounding box coordinates and region identifiers into the prompts and by normalizing model answers to match the evaluation format. Their solution targets the Physical AI Spatial Intelligence Warehouse dataset introduced in Track 3 of the 2025 AI City Challenge.

Key Contributions

1. A spatial question answering framework tailored to large-scale 3D industrial environments, using spatially-informed prompts and grounded visual cues.
2. A prompt augmentation method that embeds object-level geometric features, including bounding box coordinates and mask dimensions, into the input prompts, with each box assigned a region ID by its rank in the segmentation mask list.
3. An extension of the SpatialBot architecture fine-tuned on the Physical AI Spatial Intelligence Warehouse dataset across four spatial reasoning tasks: Distance Estimation, Object Counting, Multi-choice Grounding, and Spatial Relation Inference.
4. An output normalization module (Algorithm 1) that appends a templated suffix, "In short, the normalized answer is [label]", to training responses so predictions align with the evaluation protocol.
5. A final score of 73.0606 on the public leaderboard, placing 4th in Track 3 of the AI City Challenge 2025.

Main Findings

• Leaderboard placement: The pipeline achieved a final score of 73.0606, placing 4th overall on the public leaderboard. The top five were UWIPL_ETRI (95.8638), HCMUT.VNU (91.9735), Embia (90.6772), MIZSU (Ours, 73.0606), and HCMUS_HTH (66.8861).
• Model comparison: SpatialBot achieved an S1 score of 73.06 versus 31.92 for Qwen-VL-2.5. Per-category scores for SpatialBot were Count 78.81, Count RMSE 0.2320, Distance 46.95, Distance RMSE 1.3380, Left-Right 99.70, MCQ 66.78, Quantitative 63.2565, and Qualitative 83.9703. Qwen-VL-2.5 scored 37.96, 0.763, 13.30, 3.643, 62.02, 14.42, 25.92, and 39.28 respectively.
• Error rates: SpatialBot recorded low error rates for Count RMSE (0.2320) and Distance RMSE (1.3380), compared with 0.763 and 3.643 for Qwen-VL-2.5.
• Bounding box grounding ablation: SpatialBot_v1 without bounding box grounding scored 47.69 on S1; SpatialBot_v2 with explicit grounding scored 73.06, a substantial improvement attributed to spatial grounding aligning visual object regions with natural language queries.
• Weakest category: Distance estimation remains the lowest-performing category for SpatialBot (46.95) among the reported task metrics.
• Qwen-VL-2.5 limitation: The authors hypothesize its weak distance performance stems from the base model expressing depth using depth points rather than meters, as used in the training set; they note this discrepancy would require additional training to overcome.
• Qualitative examples: In the counting example, the model identifies Region 14 as the rightmost shelf and returns the normalized answer "3", matching the ground truth count; the figure's ground truth text names Region 0 as the closest buffer while the reported prediction names Region 1. In the pairwise comparison example, the model answers "left", matching the ground truth.

Methodology in Plain English

The researchers start from SpatialBot, a vision-language model that takes both RGB images and depth maps so it can reason about 3D layout. Input images are resized to 384 × 384 to match the pretrained encoders, and depth is encoded into a three-channel uint8 format. The original SpatialBot work compared Phi-2 (3B), Qwen-1.5 (4B), and LLaMA-3 (8B) backbones; the authors adopt Phi-2 for its balance of performance and size.

The main idea is prompt enrichment. Instead of a vague question such as "Is the pallet <mask> to the left or right of the pallet <mask>?", the modified prompt states the bounding box sizes in x1y1x2y2 form and names each object region by ID, for example "Is the pallet Region 0 within bounding box (139.2, 160.0, 160.6, 205.8) to the left or right of the pallet Region 1 within bounding box (222.8, 296.5, 253.4, 353.7)?" This gives the model concrete geometry to reason over rather than relying on appearance alone.

Because the model's free-form answers are descriptive while evaluation expects a short canonical answer such as "left" or "3", the authors append the suffix "In short, the normalized answer is [label]" to every training response. At inference, a normalization step extracts the substring after that phrase, removes punctuation, lowercases it, or falls back to matching spatial cues such as "left", "right", or "meters".

Fine-tuning used 100,000 prompts randomly sampled from the roughly 500,000 available instances, run for 12,500 iterations with the AdamW optimizer, a learning rate of 2 × 10⁻⁴, weight decay of 0.01, and a batch size of 8. LoRA fine-tuning with rank 128 and alpha 256 was applied to reduce memory and time. On 2 NVIDIA A40 GPUs (48GB each), training time per epoch was approximately 127 hours.

Why This Matters

The work shows that explicit geometric cues in prompts, combined with answer-format normalization, can meaningfully improve spatial reasoning in cluttered industrial scenes, and the ablation (47.69 to 73.06 S1) isolates bounding box grounding as the decisive factor. It also documents practical costs: training time per epoch of roughly 127 hours even with LoRA, and about 12 hours to run inference over the 19,000-image test set on a 16 GB GPU with 64 GB RAM, which the authors note limits broader adoption and experimentation.

Real-world applications:

• Warehouse navigation and autonomous material handling, where agents must judge relative positions of pallets, shelves, and buffer zones.
• Inventory management and counting, using region-grounded queries to count objects within specified zones.
• Safety monitoring, where understanding object layout and distances supports hazard detection in cluttered spaces.
• Robotics manipulation and logistics automation requiring grounded spatial relations, such as identifying whether one object is left of or right of another.

Industry relevance: the dataset and task come from an AI City Challenge track built on NVIDIA tooling (Omniverse for data creation, IsaacSim for object labels and region masks), indicating direct relevance to industrial simulation and logistics-scale deployments. The reported resource demands also serve as a realistic signal of what fine-tuning and inference of depth-augmented VLMs currently require.

Future Directions

• Reducing training cost, since the authors report roughly 127 hours per epoch even with LoRA and pretrained weights, which they say restricts experimentation with different LLM architectures.
• Reducing inference latency: running the 19,000-image test set took about 12 hours on a 16 GB GPU and 64 GB RAM, limiting the ability to try alternative approaches under competition time constraints.
• Improving distance estimation specifically, the lowest reported category at 46.95, and addressing the base model's use of depth points rather than meters through additional training.
• Testing whether the prompt-enrichment and answer-normalization strategy transfers to other VLM backbones and to industrial environments beyond the warehouse setting.

Target Audience

Researchers and practitioners working on vision-language models, spatial reasoning, and embodied or robotic perception; engineers applying depth-enhanced VLMs to industrial and logistics settings; and competition participants interested in the AI City Challenge Track 3 task, its dataset scale, and its resource requirements.

Authors’ abstract

Spatial reasoning in large-scale 3D environments such as warehouses remains a significant challenge for vision-language systems due to scene clutter, occlusions, and the need for precise spatial understanding. Existing models often struggle with generalization in such settings, as they rely heavily on local appearance and lack explicit spatial grounding. In this work, we introduce a dedicated spatial reasoning framework for the Physical AI Spatial Intelligence Warehouse dataset introduced in the Track 3 2025 AI City Challenge. Our approach enhances spatial comprehension by embedding mask dimensions in the form of bounding box coordinates directly into the input prompts, enabling the model to reason over object geometry and layout. We fine-tune the framework across four question categories namely: Distance Estimation, Object Counting, Multi-choice Grounding, and Spatial Relation Inference using task-specific supervision. To further improve consistency with the evaluation system, normalized answers are appended to the GPT response within the training set. Our comprehensive pipeline achieves a final score of 73.0606, placing 4th overall on the public leaderboard. These results demonstrate the effectiveness of structured prompt enrichment and targeted optimization in advancing spatial reasoning for real-world industrial environments.