Where Do Vision Language Models Fail at 3D Reasoning?

AI summary

Overview

• Research area: Computer vision and multimodal evaluation — specifically, diagnosing where vision-language models fail at 3D and spatial reasoning.
• Technical level: Advanced (familiarity with VLMs, benchmarking, linear/attention probes, and LoRA fine-tuning helps, though the findings are stated plainly).
• Scope: The paper introduces BedBench, a 13-task procedurally generated 3D benchmark, and uses it with twelve open-weight and four frontier VLMs plus 30 human participants to separate failures of perceiving a property, using it consistently, and depending on the image at all.

What This Paper Is About

Vision-language models often answer individual questions about a 3D object correctly — counting blocks, judging whether it fits in a box — yet fail when asked for a single exact answer that requires every quantity and attribute to be right at once. The authors build a controlled benchmark of voxel objects and geometric scenes to measure that gap precisely, and then run diagnostics to locate whether the failure comes from the visual features, from how the language model uses them, or from the model never needing the image in the first place.

Key Contributions

1. BedBench, a benchmark of thirteen procedurally generated tasks ("Blind 3D Benchmark") built on voxel objects of 10–15 connected unit blocks rendered from two opposite views, plus scenes of 6–8 collision-free boxes, cubes, and cylinders for the mixture-of-shapes task. The generator provides exact ground truth and filters out ambiguous samples at generation time.
2. Linked-question design with consistency checks, where answers across tasks entail one another on the same object (the per-color counts in T5 determine the distinct-color count in T4 and the most-common colors in T7), so a model's answers can be tested for agreement rather than only for accuracy.
3. Diagnostic controls applied to the same objects: image-removal tests, matched 2D-versus-3D renderings, factorized single-color querying, counts-supplied-as-text prompting, a fixed-layout multiple-choice format, and LoRA fine-tuning of Qwen3.5-27B on 800 in-distribution examples with the vision encoder frozen.
4. Frozen-feature probes across 14 encoders covering tasks T1–T10, using five-fold cross-validation grouped by object, plus attention-readout probes trained on 800 generated objects to test whether exact per-color counts are recoverable from visual representations.

Main Findings

• Exact per-color counting collapses. On T5: Color Distribution, humans reach 93.3% accuracy while the best VLM reaches 7.2%. Averaged over all tasks, humans reach 70.0%, the strongest model (Gemini-3-Flash) 51.5%, and the best-constant baseline 36.7%.
• Some individual judgments exceed human performance. The best models exceed the human 95% confidence interval on T2: Packaging and T6: Local Structure; on T6 the best model's 90.0% also exceeds the 77.0% obtained by always giving the most frequent answer.
• The constituent counts are already unreliable. Across the 12 open-weight models in Non-Thinking mode, per-color mean absolute error is 0.80 blocks over the full palette — worse than the 0.69 obtained by always predicting each color's mean count, though five models including the two strongest outperform that baseline.
• Simplifying the output does not help. Asking twelve independent single-color questions and merging the results changes accuracy for present colors only from 32.5% (joint) to 33.2% (factorized), and the merged predictions achieve 0.0% full-distribution exact match across all 12 open-weight models.
• 3D presentation itself is a major source of difficulty. For Qwen3.5-27B, exact match rises from 3.0% in 3D to 52.4% when the same blocks are shown as separate 2D tiles (+49.4 pp; 95% CI [+39.8, +59.0]), and per-color MAE drops from 0.50 to 0.11. After merging the frequently confused gold and yellow categories, exact match reaches 89.8% in 2D versus 6.0% in 3D.
• Fine-tuning teaches counting but not consistent use. LoRA fine-tuning lifts Qwen3.5-27B's T5 exact match from 3.0% to 68.6%, yet the same model answers T7: Color Dominance from images at only 35.0%, rising to 74.2% when given its own T5 counts as text (+39.2 pp; 95% CI [+26.8, +51.4]). When those counts are exactly correct, text-conditioned accuracy reaches 99.4% versus 40.8% for direct visual questioning.
• Inconsistency is widespread, not a fine-tuning artifact. Per object, the direct T7 answers of the 12 open-weight models match the most frequent colors in their own counts on only 21.6% of objects (chance 2.4%), and their direct T1 answers match the sum of their own counts no more often than chance.
• Reasoning helps selected tasks but leaves exact answers near floor. At the default 1024-token budget, the largest gains are on T2: Packaging (+13.8 pp) and T4: Color Count (+13.9 pp), while T5 remains at 0.7%–1.0% and T3: Difference also stays near floor; accuracy declines on T1: Volume and T9: Visible Floors.
• Larger reasoning budgets can hurt. For Qwen3.5-27B and Gemma4-31B, T4 rises from 49.0% without thinking to 59.2% at 2048 tokens and then levels off, but every larger budget leaves T1 below the budget-0 level (37.4% to 19.8% at 4096 tokens), and budgets of 2048 tokens or more do the same on T11: Projection (51.5% to 45.8%).
• Reasoning often extends a misreading. In an exploratory manual audit, visual misinterpretations occur in 42 of the 44 annotated reasoning traces that end in incorrect answers.
• Scaling helps unevenly. Within Qwen3.5 in Non-Thinking mode, growing from 0.8B to 27B improves six tasks (T2, T4, T6, T7, T9, T11) by 18–42 pp with paired bootstrap intervals excluding zero, while the remaining seven tasks change by at most 6.4 pp in point estimate; both Qwen3.5 and Gemma4 stay near floor on T3 and T5. Mixture-of-experts variants average 43.1% (Qwen3.5-35B-A3B) and 38.5% (Gemma4-26B-A4B) in Non-Thinking mode, below dense counterparts of similar total size (46.8% and 46.5%), and both remain at 0.0% on T3 and T5.
• Visual cues help only selectively. Adding color helps most on T12: Reconstruct (+6.8 pp) and raises T3 from 0.2% to 3.7%, while on T5 coordinate-axis overlays provide no improvement, with accuracy decreasing from 0.7% to 0.5%.
• Pooled probes on averaged features rarely recover exact counts. Trained on the 100 benchmark objects, probes reach at most 3.5% exact match across encoders, including a depth-supervised one — mirroring the VLM results.
• With more training data, the features do hold the counts. Attention probes using 800 generated training objects recover exact counts from frozen Qwen3.5-27B and Gemma4-31B vision-encoder tokens on 61.7% and 44.7% of objects, versus 3.0% and 4.0% for the VLMs themselves; the same probe design reaches 65.0% on Qwen3.5-27B's connector output, the image embeddings supplied to its language model, against 3.0% for the model.
• The bottleneck is in using the information, not perceiving it. Training only the language model (encoder and connector frozen) raises T5 to 68.6%, Qwen3.5-27B counts far more accurately with 2D tiles, and aggregate-quantity probe scores stay flat across encoder sizes while VLM accuracy rises.
• Some scores survive without any image. With images removed, all three tested models answer 0 on every T10: Color Connected instance and 3 on every T9: Visible Floors instance, reaching 51% and 43% accuracy respectively.
• Answer layout alone can inflate scores. Posing T1: Volume as a five-option multiple-choice question with the correct volume always the third, middle option raises accuracy by +39.0 pp on average across open-weight models (up to +57.8 pp per model); without images, Gemma4-31B reaches 100% by always choosing the third option. With shuffled candidate positions, always choosing one position yields at most 37.3% on T11 and T13 and 51.0% on T12.
• Generation-time filtering is audited. Excluding the one object with a block occluded in both views shifts model accuracies by at most 1.0 pp and changes no findings.

Methodology in Plain English

The authors generate synthetic 3D objects from unit cubes and render each one from two opposite viewpoints, so every question has an exact, machine-known answer. They discard objects that look ambiguous — for instance, objects whose two silhouettes match some other object with different orthographic projections, or objects containing a block whose removal barely changes both silhouettes (IoU ≥ 0.9998). Tasks are deliberately chosen so that producing the answer is hard but checking it is easy, and each color task's answer logically entails another's, which lets the authors test whether a model's answers agree with each other.

They evaluate twelve open-weight models from four families (Qwen3.5, Gemma4, GLM-4.6V-Flash, Kimi-VL-A3B) and four frontier models (GPT-5, Grok-4.20, Gemini-3-Flash/3.1-Pro) on 100 instances per task, except T11 and T13 which use 300 questions covering three projection planes for 100 objects or scenes. Open-weight models are run with greedy decoding in Non-Thinking and Thinking modes at a 1024-token default budget, with five repeats per condition; frontier models are queried once in Thinking mode through provider APIs. Thirty volunteers each answer one random instance per task as a human reference, reported with 95% Wilson confidence intervals.

To localize the failure, the authors intervene in four ways: they simplify the output into twelve single-color questions, they re-render the same blocks as separate 2D tiles, they fine-tune only the language model on 800 examples, and they hand the model its own counts back as text. They also train small probes on frozen encoder features to see whether the needed information is present in the visual representation, and they strip the images entirely to check whether correct answers actually depended on them.

Why This Matters

Impact on research. The paper argues that 3D benchmark scores should be reported with exact-answer correctness, agreement between logically linked answers, and evidence that scores depend on the image. It provides a concrete demonstration that accuracy alone can overstate 3D understanding — a model can recall a count in its features yet answer inconsistently, or score well by exploiting a fixed answer layout.

Real-world applications:

• Robotics and manipulation, where a system must report exact per-object counts and colors before grasping or packing, not just make plausible individual judgments.
• Warehouse and logistics automation, where fit-in-container and stacking decisions (the T2: Packaging and T6: Local Structure style questions) already work well but exact inventory-style counts do not.
• Assistive and industrial inspection systems that describe a physical object or scene from photographs, where partial correctness on a parts list is not usable.
• Evaluation and procurement of multimodal models, where buyers need to know whether a benchmark gain reflects visual ability or an artifact of answer formatting.

Industry relevance. The fixed-layout multiple-choice result is directly relevant to anyone shipping or auditing VLM evaluations: a +39.0 pp average gain, and 100% without images in one case, came from the position of the correct option rather than from seeing the object. The probes-on-frozen-features result also matters for model design, since it indicates that encoder-side information is available and the loss happens in the language model's use of it.

Future Directions

• Training objectives that enforce consistency, such as training on exact labels across many tasks, requiring models to state intermediate structure before answering, and penalizing disagreement between related answers.
• Deeper internal diagnostics, including probing language-model layers (token-level and hidden-state probing) to explain why available visual information is not used reliably — the authors note that such analyses were run on representative model families and selected tasks rather than the full suite.
• Extending the diagnostics beyond synthetic voxels to natural and embodied scenes with noisy textures, lighting variation, and complex geometry, and broadening the 30-participant human baseline to more diverse cohorts.
• Separating the causes behind the 3D penalty, since the 2D-tile control shows a large gap but does not by itself isolate occlusion, view integration, and layout effects.

Target Audience

Researchers and engineers working on vision-language models, multimodal benchmarking, and spatial or 3D reasoning; evaluation leads who design or audit model benchmarks; and practitioners choosing models for tasks that require exact counts, attributes, and part relationships rather than approximate descriptions. Readers with some background in model evaluation will get the most from the diagnostic design, though the central findings are stated without requiring the probe and fine-tuning details.

Authors’ abstract

Vision-language models can get individual judgments right while struggling to recover and consistently use the exact properties of a 3D object. To characterize this gap, we introduce BedBench, a controlled benchmark of thirteen procedurally generated tasks spanning individual judgments and composite answers. Across twelve open-weight and four frontier models, per-color counting exposes a pronounced weakness: humans achieve 93.3% accuracy, compared with 7.2% for the best model. Targeted fine-tuning raises one model's accuracy from 3.0% to 68.6%, but reveals a further difficulty: its answers about the most common colors remain inconsistent with its own per-color counts. Supplying those counts as text improves its judgments, showing that recovering a property does not mean using it consistently. Reasoning, scaling, and added visual cues each improve individual judgments, but none makes exact answers reliable. To locate the difficulty, we probe Qwen3.5-27B: exact per-color counts remain recoverable at the language model's input, yet the model's answers rarely recover them (65.0% for the probe vs. 3.0% for the model). Image-removal controls further show that some apparent successes require no visual evidence. These findings motivate evaluating 3D reasoning by exact correctness, consistent use, and image dependence, rather than accuracy alone. Code and dataset are publicly available: https://github.com/leduckhai/BedBench