RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment

AI summary

Overview

Research area: AI evaluation — 3D spatial reasoning and closed-loop visual control in vision-language models (VLMs), positioned at the intersection of multimodal benchmarking, cognitive-science-inspired spatial perception, and embodied/interactive agent evaluation.

Technical level: Intermediate. The core idea is easy to grasp (rotate a shape until its shadow matches a target), but the paper's diagnostics rely on SO(3) pose error, symmetry equivalence classes, projection IoU, and stratified sampling, so parts of Sections 3.2–3.4 and the appendices are Advanced.

Scope: The paper introduces RoTA-Bench, an automatically constructed, solvability-guaranteed benchmark of 674 interactive 3D-to-2D projection-alignment tasks (with a frozen stratified 100-task test set), and reports a large evaluation of nine VLMs, two agent frameworks, greedy/stochastic baselines, and 20 human participants.

What This Paper Is About

The authors ask whether vision-language models can convert 2D observations into 3D understanding precise enough to drive action — inferring an object's geometry, anticipating how rotation will change its appearance, and correcting course using visual feedback. Existing benchmarks either ask static questions about fixed images (no action, no feedback) or test interactive agents on tasks dominated by planning, coding, or grid-based pathfinding rather than 3D orientation. RoTA-Bench fills that gap with a simple but demanding loop: a model sees a target silhouette and a rotated 3D object, then must pick successive rotations so the object's projected silhouette matches the target within a limited action budget.

Key Contributions

1. A new interactive benchmark formulation. The paper casts 3D orientation reasoning as closed-loop projection alignment: each task provides an invariant target silhouette, the current projection, and a depth-shaded auxiliary view, and the agent selects from 24 discrete rotation actions (three camera axes — pitch, yaw, roll — two directions each, at step sizes of 1°, 5°, 10°, and 30°) plus a STOP command. Success is projection IoU ≥ 0.90.

2. An automated, solvability-guaranteed construction pipeline. 3D objects are built by extruding 2D vector icons from the open-source Game-icons library inside a sphere-bounded envelope, which preserves the icon's contour in the canonical front view and keeps every rotated pose within the viewport. Initial poses are generated by sequences of 6 to 16 reversible 30° rotations, so a valid solution exists by construction for every task. The pipeline yields 674 canonical tasks (674 unique source icons: 641 CC BY 3.0, 33 CC0) and a 100-task test set chosen by proportional stratified sampling (Hamilton's method) over reference path lengths of at least 4 steps, with the remaining 574 tasks reserved for future training and research.

3. Reference solutions and a decomposed diagnostic metric suite. Reference paths come from pose-guided beam search and 30°-restricted breadth-first search, which find substantially shorter trajectories than the raw reversed generation paths. Beyond task success, the paper measures final IoU, executed steps, surface coverage, Effective Direction Rate, Best Legal Angle Rate, Angle Gap, and Angle Loss — separating which way a model rotates from how far.

4. A controlled human comparison and failure diagnosis. Twenty undergraduates each completed a consecutive 20-task block of the test set under an identical interface, action vocabulary, and stopping rule, producing 400 human task runs. The authors use matched states between humans and GPT-5.6-Sol to show where image-space and pose-space progress diverge.

Main Findings

• A large human–model gap. GPT-5.6-Sol has the best model success rate at 17.0% (final IoU 0.598); Gemini 3.6 Flash reaches 10.0%, GPT-6-Astra 10.0%, GPT-5.6-Terra 8.0%, GPT-5.6-Luna 4.0%. Claude Sonnet 5 (0.492), Claude Opus 5 (0.484), and Qwen 3.7 Plus (0.504) reach 0.0% success with final IoU barely above uniform random exploration (0.473). Humans reach 100.0% success and mean final IoU 0.923 in 16.22 steps. By contrast, the pose-greedy oracle baseline (given ground-truth 3D pose error) solves 100.0% in 8.32 steps at IoU 0.924, while the IoU-greedy baseline solves only 39.0% (IoU 0.725) — showing that greedily maximizing visible overlap is insufficient.

• Agent wrapping does not rescue performance. ReAct reaches 16.0% success with the highest model final IoU (0.628) over 24.62 steps; Magentic reaches 2.0% (0.529) in 16.00 steps. An IoU-rollback random baseline reaches 0.7% (0.545) while consuming 35.57 steps.

• Policy collapse in a spatially specialized model. SpatioLM-Understanding-InternVL3.5 fails entirely (0.0% success, final IoU 0.463, below uniform random), with 91.12% of its actions degenerating into YAW_POS 10°.

• Coverage is not the bottleneck. GPT-5.6-Sol's cumulative surface coverage (79.58%) closely trails humans (83.07%), and ReAct (79.08%) and GPT-6-Astra (78.93%) are comparable — so failure is not explained by looking at too little of the object.

• Direction and angle are both deficient. GPT-5.6-Sol selects an effective rotation direction 58.75% of the time versus 83.91% for humans, and only marginally above uniform random (53.78%). Conditional on an effective direction, its chosen angle deviates 6.49° from the best legal angle versus 2.99° for humans, with a Best Legal Angle Rate of 61.80% versus 71.82%, and an angle loss of 2.25° per step versus 0.96°.

• Image-space and pose-space progress diverge at matched states. Across 211 exact state pairs shared by humans and GPT-5.6-Sol (prior coverage 42.01% vs. 46.21%), the model executes nearly twice as many "IoU up, pose error up" actions (19.43% vs. 10.90%) — the 19.4% vs. 10.9% figure in the abstract. Humans execute more "IoU down, pose error down" actions (33.18% vs. 23.70%) and more "both improve" actions (50.24% vs. 34.60%), and far fewer "both worsen" actions (5.69% vs. 22.27%).

• Models key off visible overlap more than true 3D discrepancy. GPT-5.6-Sol's rotation-angle choices correlate more strongly with pre-action IoU (ρ = −0.536, 95% CI [−0.617, −0.443]) than with pose error (ρ = 0.368, 95% CI [0.166, 0.532]). Humans are strongly associated with both (ρ = −0.696, 95% CI [−0.763, −0.633]; and ρ = 0.751, 95% CI [0.675, 0.828]). Most evaluated models show the same relative pattern as GPT-5.6-Sol.

• Fine-grained control depends on interaction history. In a 30-task ablation (a subset whose reference-step distribution matches the full test set: Spearman ρ = 0.9121, p = 3.56 × 10⁻⁵, bootstrap 95% CI lower bound 0.7445), full input gives 16.67% success and final IoU 0.5656. Removing the auxiliary depth view drops success to 3.33% and pushes 86.14% of actions to the 30° step. Removing interaction history drops observed success to 0/30 and final IoU to 0.4822 — near uniform random (0.4808) — with 30° actions rising to 96.76% from 62.75%, 1° adjustments falling to 0.16% from 12.18%, and the IoU–angle correlation weakening to ρ = −0.242. Effective-direction rate barely moves across ablations (54.85%–56.52%): history mainly informs how far to rotate, not which way.

• Task difficulty tracks reference path length. For GPT-5.6-Sol, tasks with 4–6-step reference solutions yield 12/55 successes (21.8%), versus 5/45 (11.1%) for 7–16-step tasks.

Methodology in Plain English

The authors built a puzzle. Take a flat icon (say, a T-shirt), give it thickness so it becomes a 3D solid but keep it inside a sphere, and you get an object whose front-on shadow exactly reproduces the original drawing. Show the solver that target shadow, then present the same object already twisted into some random orientation (generated by stacking 6–16 random 30° turns). The solver picks rotations one at a time from a fixed menu of 24 options plus "stop," watches how the shadow changes after each move, and tries to get the projected silhouette to overlap the target by at least 90% (measured as intersection over union) before using up 48 decisions or stalling.

Because the starting orientation was made by real rotations, undoing them always works — so every task is guaranteed solvable, and a separate search (beam search plus a breadth-first search restricted to 30° steps) finds shorter reference solutions. One hundred tasks were frozen as the test set, stratified by how many reference steps each needs.

For evaluation, everyone — nine VLMs queried at temperature 0 via APIs (total cost $1,482.07), two agent wrappers, and 20 undergraduates clicking buttons — faced the identical observations, action tokens, thresholds, and stopping rules. SpatioLM was deployed locally on eight NVIDIA A100 (40GB) GPUs. Crucially, the researchers also recorded ground-truth 3D poses (hidden from the models) so they could score pose error separately from visible silhouette overlap, using a symmetry-aware rotation distance, and could classify each action as helping or hurting in image space versus pose space. The human study collected no personally identifiable information; 99% of the 400 human runs finished within 278.12 seconds (4.64 minutes) per task.

Why This Matters

Impact on research. The paper separates two things benchmarks often conflate: recognizing or matching an image, and converting visual evidence into precise 3D control. Its matched-state analysis shows that a model can look better in image space while moving further from the correct 3D pose — a failure mode that image-similarity metrics alone would hide. The ablation result, that the effective-direction rate is nearly unchanged while angle selection collapses without history, localizes a specific deficiency (magnitude estimation) rather than a generic "poor spatial reasoning" verdict. Because the pipeline is automated and solvability is guaranteed by construction, it also offers a scalable environment for training, not just testing.

Real-world applications:

• Robotic manipulation and bin picking, where a gripper must rotate a part until its silhouette matches a target pose under occlusion.
• Augmented-reality alignment, where a virtual object must be rotated to register against a camera view.
• Inspection and assembly verification, judging whether a part's projected outline matches a reference from a single monocular camera.
• Embodied agents and vision-language-action systems that must act on orientation under a limited step budget, rather than issuing long plans.

Industry relevance. The gap between "cheap" image-space similarity signals and true geometric correctness matters for anyone deploying VLMs in perception-and-control loops: optimizing IoU-style objectives can be actively misleading, as the 39.0% IoU-greedy success rate demonstrates. The SpatioLM policy collapse (91.12% of actions reduced to one token) is also a concrete warning about degenerate action distributions in spatially specialized models. The total API cost and single-GPU-server deployment for the open model indicate the harness is inexpensive to reproduce relative to human studies.

Future Directions

1. Richer shapes and scenes. The authors note that extruding from a limited set of 2D icons constrains shape diversity and cannot represent complex 3D geometry; they propose synthesizing icons with generative models or building more expressive 3D meshes directly.

2. Harder interaction dynamics. Each task has a single target under discrete rotation actions. Extensions suggested include continuous rotation control, external occlusions, and multi-object alignment or assembly tasks.

3. Training rather than only testing. The automatically generated tasks are proposed as a scalable environment for reinforcement-learning fine-tuning, to test whether interactive training improves the use of visual feedback for precise 3D control.

4. Broader human baselines. The human study used 20 undergraduates; the paper states that future work will evaluate participants beyond the undergraduate population. Whether the 100% human success rate and the 2.99° angle gap hold for other populations is not reported here.

Target Audience

Researchers and engineers working on multimodal model evaluation, spatial and 3D reasoning, and embodied or interactive agents — particularly those deciding which metrics to trust when a model's visible output looks right. It is also useful for practitioners deploying VLMs in perception-and-control pipelines, and for benchmark designers interested in automated task generation with solvability guarantees and in the IoU-versus-pose diagnostic separation. Readers wanting only the headline result can read the abstract and Section 5.1; readers reproducing or extending the benchmark will need the appendices on geometry construction, search references, prompts, and statistics.

Authors’ abstract

Can vision-language models turn 2D observations into 3D understanding that supports precise action? This requires inferring object geometry, anticipating the effects of rotation, and refining actions through visual feedback. We introduce RoTA-Bench to evaluate these abilities through interactive 3D-to-2D projection alignment. Given a target silhouette, a model observes a rotated object and chooses successive rotations to recover the target projection within a limited action budget. Our automated pipeline constructs silhouette-preserving 3D objects from 2D icons and uses reversible initialization to guarantee solvability, yielding 674 tasks and a stratified 100-task test set. Empirically, we find that (i) a substantial human-model gap persists: GPT-5.6-Sol succeeds on only 17% of tasks, whereas 20 undergraduate participants collectively complete the test set four times with a 100% success rate; (ii) image-space and pose-space progress can diverge: at matched states, the model more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%); and (iii) fine-grained control is sensitive to interaction history: removing it leaves 96.8% of the model's actions at a coarse 30-degree step size and sharply reduces observed task success. These findings highlight the challenge of converting visual evidence into precise 3D control, beyond recognizing or matching individual images.