BOP-ASK: Object-Interaction Reasoning for Vision-Language Models

AI summary

Overview

Research area: Computer vision and embodied AI — specifically vision-language models (VLMs), 3D spatial reasoning, and robot manipulation. The work sits at the intersection of pose estimation benchmarks (the BOP family) and VLM training/evaluation.

Technical level: Intermediate. The paper is readable without deep robotics background, but familiarity with terms like 6D object poses, grasp affordances, motion planning, and VLM fine-tuning helps.

One-sentence scope: The paper introduces BOP-Ask, a large-scale dataset and pair of benchmarks that test whether VLMs can reason about how objects physically interact — where to grasp them, how to move between them, and what must be moved out of the way first.

What This Paper Is About

Existing spatial reasoning benchmarks for VLMs test coarse relationships such as "left of" or "behind," but they say nothing about the fine-grained physical understanding a robot needs: precise 3D localization, physical compatibility between objects, affordances, and multi-step spatial planning. The authors address this by building a dataset grounded in accurate 6D object poses from the Benchmark for Object Pose Estimation (BOP), from which they automatically derive grasp poses, motion trajectories, relative spatial and depth relationships, and object-to-object relationships. The goal is a single resource that can both train VLMs for object-interaction reasoning and benchmark them against human performance.

Key Contributions

1. An object-interaction reasoning dataset built on BOP. The authors augment existing perception data with queries about object interactions, manipulation affordances, motion feasibility, and scene-level reasoning, producing over 150k images and 33M question-answer pairs across six tasks (four of them novel).

2. Two new evaluation benchmarks. BOP-Ask-core is drawn from hand-selected held-out BOP scenes (688 VQA pairs, manually verified), and BOP-Ask-lab is hand-constructed from images taken in different labs, making it fully independent of BOP in terms of camera, objects, and point of view (240 VQA pairs from 15 images).

3. Demonstrated gains from training on BOP-Ask. Fine-tuned VLMs improve not only on the paper's own test sets but also on out-of-domain spatial reasoning benchmarks — RoboSpatial-Home, SpatialBench, and CV-Bench — and show emergent capabilities such as precise object and grasp pose estimation, trajectory planning, and object-centric spatial reasoning in cluttered environments.

4. A human baseline and real-robot validation. Human participants were evaluated on BOP-Ask-core using the same metrics as the models, and a fine-tuned model was deployed on physical pick-and-place tasks.

Main Findings

• Existing VLMs struggle badly. On BOP-Ask-core, GPT-5 scores 9.0 3D IoU on pose estimation, 0 success rate on trajectory prediction, and 14.8% recall on object rearrangement. Several models produce no valid grasp output at all ("inf" in the results table).

• Fine-tuning on BOP-Ask produces large gains. NVILA (15B) fine-tuned reaches 73.5 3D IoU on pose, 64.2 success rate on trajectories, 77.4 distance error, 1.40 NCE on grasps, 95.8 spatial success, 94.6 relative-depth success, and 57.7% rearrangement recall. NVILA (2B) fine-tuned reaches 77.4 pose IoU, 50.8 trajectory success, 78.5 distance error, 1.69 NCE, 94.2 spatial, 94.6 depth, and 56.4% rearrangement recall. Qwen-VL 2.5 (3B) fine-tuned reaches 48.2 pose IoU, 22.5 trajectory success, 116.3 distance error, 1.5 NCE, 92.6 spatial, 94.1 depth, and 43.4% rearrangement recall.

• Humans are better but not perfect. The human baseline on BOP-Ask-core scores 54.2 3D IoU on pose, 67.3 trajectory success, 112.4 distance error, 1.1 NCE on grasps, 84.9 spatial success, 87.3 relative-depth success, and 44.1% rearrangement recall — indicating that the tasks are genuinely hard even for people, particularly pose, grasp, and trajectory prediction.

• Object rearrangement remains the hardest task. The best-performing models only reach about 60% accuracy, which the authors attribute to the need for fine-grained object-object relationships, 3D coordinate alignment, and clutter dynamics.

• Binary spatial and depth tasks are relatively easy. These tasks show higher accuracies because they reduce to comparative judgments aligned with relational priors acquired during pre-training.

• Transfer to out-of-distribution benchmarks. Fine-tuning improves NVILA from 63.4 to 69.1 on RoboSpatial-H, from 78.2 to 89.3 on CV-Bench, and from 47.5 to 50.0 on SpatialBench. Qwen-VL 2.5 improves from 78.1 to 81.3, 88.8 to 92.4, and 60.0 to 65.0 respectively. On BOP-Ask-lab, NVILA improves from 6.1 to 16.2 pose IoU, 4.2 to 1.1 grasp NCE, 0 to 28.2 trajectory success, and 70.0 to 81.2 spatial-depth accuracy; Qwen-VL 2.5 improves from 12.6 to 25.3, 3.6 to 1.3, 0 to 37.1, and 74.4 to 85.8.

• Data diversity helps. Ablations adding YCB-V, HANDAL, and LINEMOD progressively improve NVILA: pose IoU rises from 6.5 (base) to 31.7, to 54.4, to 67.2, to 77.4 with the full BOP-Ask. Trajectory success goes from 0 to 24.2 to 30.8 to 51.8 to 64.2.

• Removing the yes/no tasks hurts. Training without spatial and depth questions drops spatial success from 94.2 to 62.5 and depth from 94.6 to 50.6, confirming these auxiliary tasks strengthen multi-skill co-training.

• Real-robot results. Across 15 pick-and-place tasks on a Franka robot, base NVILA completed none; the BOP-Ask fine-tuned NVILA succeeded on 10 out of 15.

Methodology in Plain English

The authors start from the BOP family of pose-estimation datasets — HOPE, HANDAL, YCB-V, and LineMOD — which supply RGB-D images, camera intrinsics, and ground-truth object poses. Because object orientations are a poor proxy for gravity, they reconstruct a world coordinate frame: they use a pointing-focused VLM to locate the tabletop, fit a plane to the corresponding 3D points with RANSAC, and compute the rotation aligning the world up-axis to the plane normal via the Rodrigues rotation formula.

From these poses they compute 3D cuboid bounding boxes for each object. Motion trajectories are generated with a Rapidly-exploring Random Tree (RRT) planner in 3D Cartesian space, initialized at one object's centroid with 10% goal-biased sampling toward another object; paths that intersect neighboring object meshes are discarded, and the raw paths are simplified with the Ramer–Douglas–Peucker algorithm. Grasps are computed with M2T2, a transformer-based parallel gripper model using dual sampling of global scene points and object-centric points, keeping the top-5 grasps per object. If every predicted grasp collides with surrounding objects, the object is labeled "fully cluttered" — which becomes the basis for the rearrangement task.

Question generation is semi-automatic: 3D models are rendered and sent to a VLM for descriptions of shape, color, size, and utility, which are then manually verified. Task-specific templates produce questions, which an LLM rewrites into linguistically diverse, human-like phrasing using in-context examples. To disambiguate repeated object categories, the system assigns positional attributes such as "leftmost," "rightmost," "topmost," and "bottommost." Poses, grasps, trajectories, and rearrangements are encoded as ordered 2D lists of keypoints; spatial and depth questions are binary yes/no.

Finally, the authors fine-tune Qwen-VL 2.5 and NVILA on the data, evaluate proprietary models (GPT-5, Gemini Robotics-ER 1.5) through official APIs, and collect human annotations through a simple interface. Open-source inference runs on a single NVIDIA A100 GPU; fine-tuning uses a cluster of eight A100 GPUs.

Why This Matters

Impact on research. The paper argues that existing spatial reasoning benchmarks mask critical weaknesses because they test only high-level relationships. BOP-Ask shifts evaluation toward pixel-level, executable answers rather than multiple-choice or yes/no questions, and grounds annotations in precise 3D ground truth rather than approximate monocular depth estimates. It also supplies a training corpus, not just a test set, and shows that training on it improves scores on unrelated spatial benchmarks.

Real-world applications:

• Tabletop robotic pick-and-place in cluttered environments, where a robot must decide where to grasp an object and how to reach it.
• Decluttering and rearrangement, where the system must identify which obstructing object to move first before grasping a target.
• Augmented reality and egocentric vision assistants that need to reason about physical relationships between objects in a scene.
• General embodied agents that must translate natural-language instructions into spatially grounded action.

Industry relevance. The results connect directly to warehouse automation, manufacturing bin picking, and service robotics, where a model must output grasp poses and collision-free paths rather than text descriptions. The gains from training on BOP-Ask — and the real-robot success of 10 out of 15 tasks — suggest that domain-specific 3D-grounded data is a practical lever for improving deployed VLM performance, and the released benchmarks give teams a way to measure progress on these capabilities.

Future Directions

• Object rearrangement is unsolved. With the best models at roughly 60% and humans at 44.1% recall, the authors flag fine-grained object-object relationships, 3D coordinate alignment, and clutter dynamics as key opportunities.

• Improving open-world grounded interaction. Results on BOP-Ask-lab, whose images come from different labs and cameras, show improvements but also underscore remaining challenges for generalization beyond the training distribution.

• Comparison with concurrent work. The authors note that TIGeR proposes a similar object-interaction reasoning dataset covering localization, distance estimation, and pose computation, but it was not publicly available at the time of writing, so a direct comparison on shared tasks was not feasible.

• Extending the pipeline to new scenes. The authors state that their generation algorithm can be easily extended to novel scenes with object poses, which is trivial to obtain using simulated environments.

Target Audience

This paper is most useful for robotics and embodied-AI researchers building VLMs that must act in physical environments, for dataset and benchmark designers interested in geometrically grounded annotation pipelines, and for engineers working on manipulation, grasping, or motion planning who need models that output 3D poses and trajectories rather than text. Readers focused on purely semantic vision-language tasks will find the emphasis on pose, grasp, and path metrics less directly applicable.

Authors’ abstract

Vision Language Models (VLMs) have achieved impressive performance on spatial reasoning benchmarks, yet these evaluations mask critical weaknesses in understanding object interactions. Current benchmarks test high level relationships ('left of,' 'behind', etc.) but ignore fine-grained spatial understanding needed for real world applications: precise 3D localization, physical compatibility between objects, object affordances and multi step spatial planning. In this work, we present BOP-ASK, a novel large scale dataset for object interaction reasoning for both training and benchmarking. Our data generation pipeline leverages 6D object poses from the Benchmark for Object Pose Estimation (BOP) datasets from which we derive fine grained annotations such as grasp poses, referred object poses, path planning trajectories, relative spatial and depth relationships, and object-to-object relationships. BOP-ASK comprises over 150k images and 33M question answer pairs spanning six tasks (four novel), providing a rich resource for training and evaluating VLMs. We evaluate proprietary and open sourced VLMs, and conduct human evaluations on BOP-ASK-core, a contributed test benchmark. We also release BOP-ASK-lab, an out-of-distribution benchmark with images not sourced from BOP, enabling testing of generalization. Our experiments demonstrate that models trained on BOP-ASK outperform baselines and exhibit emergent capabilities such as precise object and grasp pose estimation, trajectory planning, and fine-grained object-centric spatial reasoning in cluttered environments.