Tri-Bench: Stress-Testing VLM Reliability on Spatial Reasoning under Camera Tilt and Object Interference

AI summary

Overview

• Research area: Computer Vision / Vision-Language Model evaluation, specifically spatial and geometric reasoning robustness.
• Technical level: Intermediate.
• Scope: A compact, camera-captured benchmark of 400 images of planar triangle problems that isolates relative geometric reasoning in four recent VLMs while perturbing camera pose (planar vs. tilted) and scene context (presence of ten everyday objects).

What This Paper Is About

Vision-Language Models are increasingly proposed for agentic tasks that require trustworthy spatial reasoning, such as robot navigation, AR/VR measurement and 3D geometry teaching, yet their reliability under realistic scene changes is largely unverified. The author builds Tri-Bench, a small controlled benchmark of triangle geometry problems captured by camera, to test whether VLMs can reason about real 3D geometry when the image is tilted and when everyday objects clutter the scene. The specific question is whether models use an explicit frame-of-reference hint (a square border surrounding the triangle, which permits recovery of true geometry via homography) or simply default to measuring the 2D image plane.

Key Contributions

1. A novel controlled benchmark: Tri-Bench, 400 images built from 100 labeled triangles in a 1 meter × 1 meter square, captured in four conditions to diagnose VLM spatial reasoning robustness to camera pose and object interference.
2. A stress-test of four leading VLMs (Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-5, Qwen2.5-VL-32B) revealing a critical failure mode: models misinterpret the 3D real world as 2D projections in the image plane even when given an explicit frame-of-reference guardrail.
3. Identification of majority class bias in precision tasks, with near-zero accuracy on minority shapes and clear performance gaps across triangle shapes.
4. A reproducible, interpretable diagnostic: all code, data and scripts are released on GitHub, including a homography-based estimation procedure described in the appendix.

Main Findings

• Overall accuracy is modest and 2D-biased: Average accuracy with respect to 3D ground truth is about 69% (68.98%), with the best model at about 75% (75.30%) and the worst at about 64% (64.32%). The same responses align more closely with 2D projections in the image plane, where mean accuracy is about 72% (72.32%).
• The guardrail hint is ignored: When accuracy is recomputed against 2D image-plane projections, the Gemini models gain roughly 5–6% and GPT-5 and Qwen2.5-VL-32B improve only about 1–2%. The overall mean rises from 68.98% to 72.32%, indicating models default to 2D cues rather than using the homography information implied by the square border.
• Near-total failure on minority shape classes: Accuracy collapses to 1.88% for right-angled triangles (Q2), 1.44% for isosceles (Q1) and 0.00% for equilateral (Q1), while the majority classes look strong at 99.51% (scalene, Q1) and 85.69% (acute, Q2). Qwen2.5-VL-32B scored 0% on all non-majority shape classes.
• Q1 accuracy mirrors the dataset prior: Overall Q1 accuracy is 64.06%, almost identical to the proportion of scalene triangles in the dataset (64%), and stays near 64% across all four capture conditions P0–T1.
• Camera tilt degrades performance: Planar views (P0/P1) reach 71.0% accuracy while tilted views (T0/T1) drop to 66.9%. The abstract reports an overall degradation of about 4.1% under tilt. For Q2 and Q5 tilt reduces accuracy by roughly 6–7%, and for Q3 by about 5%.
• Object interference is negligible: No-object images (P0/T0) achieve 69.2% accuracy versus 68.8% for with-object images (P1/T1), a difference of less than 1%.
• Task-type gap: Relative comparison tasks (Q4, Q6) are much easier for all models than absolute angle identification (Q2) and specific ratio estimation (Q3); the max/min side ratio task Q5 is noticeably easier than the absolute ratio estimation task Q3.
• Model ranking is consistent: Gemini 2.5 Pro is the strongest model with Gemini 2.5 Flash close behind; GPT-5 and Qwen2.5-VL-32B consistently achieve lower accuracy.
• Shape labels shift between 3D and 2D: Across the 400 images, about 27% of triangles change side type and 34% change angle type between 3D and 2D labels. For side type, only about 7% of scalene images change class versus 62.5% for both isosceles and equilateral; for angle type, about 26% of acute, 9% of obtuse, and about 70% of right triangles change class in the image plane.

Methodology in Plain English

The author physically constructed a 1 meter × 1 meter square out of masking-tape border (tape width 4.8 cm) and placed 100 labeled triangles inside it, marking each vertex with a 3 cm × 3 cm colored paper square (A=RED, B=YELLOW, C=BLUE). The triangle set covers 38 acute, 32 obtuse and 30 right triangles, and by side type 64 scalene, 26 isosceles and 10 equilateral; the target was a 3:3:3 split of acute/obtuse/right and a 6:3:1 split of scalene/isosceles/equilateral.

Each triangle was photographed in four conditions: P0 (planar, no object), P1 (planar, with object), T0 (tilted, no object) and T1 (tilted, with object). Ten everyday objects were used, each placed in 10 positions, giving 400 images total, built from 95 uniquely shaped triangles plus 5 repetitions that perturb labels and positions.

For every image, six questions are asked about triangle ABC, covering categorical shape type (equilateral/isosceles/scalene, acute/right/obtuse) and continuous quantities (AB/AC, the difference between two angles, max/min side ratio, and the angle range). Each model receives one identical fixed zero-shot prompt that names the colored vertices and explicitly describes the light-brown masking-tape square border, then demands strict JSON output with six keys and four decimal places. No chain-of-thought or structured rationales were used, and sampling used API defaults.

Because every triangle lies in the same plane as the tape border, a single 3×3 homography per image can map the tape corners to a canonical unit square and recover side lengths and angles up to a global similarity transform, which preserves all ratios and angles. Ground truth labels thus come from the true 3D geometry, and the author also computes a 2D image-plane alternative to test which reference frame models actually use.

Scoring uses accuracy and error. Categorical questions (Q1, Q2) use 0/1 error. Ratio questions (Q3, Q5) use relative error capped at 1. Angle questions (Q4, Q6) use absolute difference normalized by 180 degrees and capped at 1. Shape labels themselves are tolerance-based: isosceles if the smallest pairwise relative side difference is at most 3%, equilateral if all pairwise relative side ratios lie within 3%, and right if some interior angle is within 2 degrees of 90 degrees.

Why This Matters

Impact on research. The paper argues that verifiability and control of agentic AI are not guaranteed by simple prompting. It shows that an explicit frame-of-reference guardrail does not make VLMs reason in 3D, and it offers a minimal reproducible diagnostic (400 images, one fixed prompt, four models) for a capability gap that broader cognitive or Olympiad-style benchmarks do not isolate. The majority class bias result also raises a concrete question about whether such bias is inherited from real-world training distributions.

Real-world applications at risk:

• Robot navigation and autonomous systems that must act on true 3D geometry rather than image-plane appearance.
• AR/VR measurement tools that report distances, angles or ratios to users.
• AI-based 3D geometry teaching, where wrong shape classifications would mislead students.
• Medical assistance and 3D reconstruction pipelines that depend on reliable spatial estimates.

Industry relevance. The author frames the findings directly for the "Trustworthy Agentic AI" community, stating that if an agent cannot perform basic 3D reasoning on a simple triangle it cannot be deployed in safety-critical robotics or autonomous navigation. The benchmark is compact and reproducible, making it usable as a lightweight reliability check for teams shipping VLM-powered measurement or perception features.

Future Directions

• Multi-view geometry: the current evaluation is based on a single image; the author suggests inference through multi-view geometry as a natural extension with minimal guardrails.
• Granular pose study: the paper tests tilt as a binary factor (planar vs. tilted); a more granular study could measure the precise relationship between specific tilt angles and accuracy degradation.
• Stronger prompting: only a single fixed guardrail prompt was used, deliberately, to test frame-of-reference following; more advanced prompting remains open.
• Beyond triangles and beyond current metrics: extensions proposed include other polygons, self-intersecting curves and smooth differentiable surfaces, alternative evaluation metrics on the released dataset, and analysis of training distributions to explain the majority class bias.

Target Audience

Researchers and engineers working on VLM evaluation, spatial and geometric reasoning, embodied AI, robotics and trustworthy agentic systems. It is also relevant to practitioners deploying VLMs in measurement, AR/VR or educational tools who need a compact diagnostic for 3D-reasoning reliability, and to benchmark designers interested in controlled, photorealistic perturbation studies.

Authors’ abstract

Verifiable geometric reasoning is a critical component for trustworthy and controllable agentic AI. Despite impressive capabilities, Vision-Language Models (VLMs) often fail under realistic scene changes. We present Tri-Bench, a compact benchmark of planar triangle problems that isolates relative geometric reasoning while stressing two deployment-critical factors: camera pose (planar vs. tilted) and scene context via object interference (10 everyday objects). To test verifiability and control, we evaluate four recent VLMs using a single, fixed prompt whose guardrail explicitly describes a surrounding square border, enabling correct answers via homography. We evaluate six simple tasks over binary and continuous targets, and observe that the overall accuracy with respect to 3D ground truth is modest, ~69% on average (best ~75%, worst ~64%). The same responses align even more closely with 2D projections in the image plane, where mean accuracy is ~72%. All four VLMs consistently fail, with accuracy falling to ~0%, on recognizing minority shape classes (equilateral, isosceles, right-angled triangles). Additionally, overall VLM accuracy degrades by ~4.1% under camera tilt. This demonstrates that models fail to correctly utilize the explicit frame-of-reference hint provided in the prompt and default to 2D image plane cues. Finally, we find that object interference has no significant effect on VLM accuracy.