When Depth Lies: Benchmarking Vision-Language Models on Mirror-Induced RGB-D Ambiguity

AI summary

Overview

Research area: Computer vision and multimodal AI — specifically spatial reasoning in vision-language models (VLMs) and the reliability of RGB-Depth sensing.

Technical level: Advanced. The paper assumes familiarity with VLM architectures, RGB-D inputs, instance segmentation, and benchmark design.

Scope (one sentence): The paper introduces MIRA-Bench, a benchmark of 1,011 RGB-D images and 14,140 question–answer pairs built to test whether VLMs can reason about space when mirrors and other reflective surfaces corrupt depth readings.

What This Paper Is About

Vision-language models that take RGB images plus depth maps are widely assumed to reason more accurately about 3D space, because depth is treated as a faithful measure of physical distance. Mirrors break that assumption: they bounce light so a depth sensor reports the round-trip path rather than the true distance, while the RGB image faithfully shows objects that are not actually there. The paper builds a dedicated benchmark to measure how badly current VLMs fail when reflections make appearance and depth disagree, and whether supplying depth helps or hurts.

Key Contributions

1. A mirror-centric RGB-D dataset. 1,011 images across 525 real-world scenes, assembled from RGBD-Mirror (406 scenes, 517 images), DVMD (34 scenes, 64 images), and 430 newly captured images across 85 scenes shot with an Intel RealSense D435. It spans indoor and outdoor settings, six reflective surface types (planar mirrors, glass surfaces, convex mirrors, vehicle side mirrors, water surfaces, and screens), and both daytime and nighttime conditions.

2. Fine-grained grounded annotations. Reflective surface region masks, class-agnostic polygon instance masks for both directly observed and reflected objects, visibility labels, and explicit real-to-reflection correspondences. Scenes are defined by reflective context rather than physical room, so the same room with a different mirror surface counts as a distinct scene.

3. MIRA-Bench: a three-level, eight-task QA benchmark. 14,140 QA pairs organized into Reflection-aware Perception (8,090 pairs, 57%), Spatial Reasoning (3,430, 24%), and Scene-level Decision-making (2,620, 19%). The benchmark also supports five different object-referencing protocols (textual bounding boxes, textual boxes plus descriptions, drawn boxes, binary masks, and drawn boxes plus descriptions).

4. The first dedicated evaluation of VLMs on mirror-centric spatial reasoning, covering ten models, plus a modality ablation showing that depth information provides no consistent benefit and can actively harm decisions.

Main Findings

• Large gap to human performance. Human annotators reached 85.0% overall with RGB and 90.7% with RGB-D. The best proprietary model, Gemini-3.8-Flash, reached 40.1%; the best open-source model, Qwen3.5-9B, reached 30.4%. Random guessing scores 14.1% overall.

• Matching works better than consistent judgment. Models scored higher on Mirror Correspondence (MC) than on Visibility Classification (VC). Because VC is scored jointly — all three related questions about an object must be correct — this gap shows models can match an individual reflected instance but cannot make stable, repeated judgments about the same object.

• Spatial inference through reflections is near-floor. Mirror-to-real spatial inference (MR) was the hardest task: Qwen3.5-9B scored 2.9%, Qwen3-VL-8B 3.2%, Qwen2.5-VL-7B 0.4%, and DeepSeek-VL2-Tiny 0.9%, against a random baseline of 3.7%. Even the best model, Gemini-2.5-Pro, reached only 22.3%.

• Depth degrades collision-risk judgment. In the modality study on Qwen3.5-9B, adding depth dropped collision risk (CR) accuracy from 52.6% (RGB) to 36.0% (RGB-D). Overall accuracy stayed in a narrow 29.6–31.3% band across all four input modes, so depth neither helps nor is ignored — it misleads.

• Mirror masks help a little, but not enough. Adding a mirror-region mask raised Mirror Correspondence from 39.8% to 42.7%. Combining mask and depth (RGB-DM) did not beat the mask alone, and MR stayed at 2.1–3.2% under every input configuration.

• Action selection is a particular weak point. GPT-5.4 scored 9.7% on Next Step Action Selection, Gemini-2.5-Pro 10.9%, Gemini-2.5-Flash 12.3%, and DeepSeek-VL2-Tiny 3.4%, against a random baseline of 16.7%.

• Referring protocol is not the bottleneck. Across five protocols on Qwen3.5-9B, overall accuracy ranged from 27.9% to 33.0% — a spread of 5.1 percentage points. Visual bounding boxes performed best (33.0%) and binary masks worst (27.9%); adding text descriptions to visual boxes gave no further gain.

Methodology in Plain English

The authors first gathered images of scenes containing reflective surfaces, mixing two existing public RGB-D mirror datasets with their own new photography. They deliberately shot outdoor and non-planar cases — convex traffic mirrors, car side mirrors, glass facades, water — because public data is dominated by indoor flat mirrors, and they captured multiple viewpoints per location since what a mirror shows changes sharply with camera position. They excluded images where reflections filled the whole frame, since even humans could not then separate real from virtual.

Annotators then outlined every reflective surface and drew polygons around every object, both the physical ones and their reflections, tagging which reflected instance corresponds to which real one. A second annotator re-labeled a random 20% of each batch of 100 images; if agreement (measured by mean Intersection-over-Union) fell below 85%, a third annotator redid the whole batch. Depth maps served as authoritative evidence for ground-truth answers about front-to-back ordering, which 2D cues cannot resolve.

Questions were generated from these annotations across eight sub-tasks and three difficulty levels: can the model tell reflected from real, can it infer real-world spatial relations through a mirror, and can it make practical judgments like reachability, next action, and collision risk. Each question was independently cross-checked by two reviewers, with a third resolving disagreements.

Finally, ten VLMs were queried at temperature 0.0 with a maximum of 2,048 output tokens, required to return JSON with a final answer, brief explanation, and confidence. Models were run in four input modes — RGB, RGB with depth, RGB with mirror mask, and RGB with both — and with five different ways of pointing at objects.

Why This Matters

Impact on research. The paper challenges a foundational assumption in RGB-D spatial reasoning: that depth is a trustworthy geometric channel. It shows that a whole class of physically common scenes silently violates that premise, and that standard benchmarks have no way to detect it. It also supplies a diagnostic structure — perception, then reasoning, then decision — that separates where a model fails rather than only reporting a single score.

Real-world applications:

• Robot and embodied-agent navigation in homes, malls, and offices with large mirrors, where a reflected open corridor could be mistaken for walkable space and glass could be mistaken for a doorway.
• Autonomous driving and parking, where convex roadside mirrors and vehicle side mirrors are common and depth sensors on vehicles must not treat reflected traffic as physically present.
• Assistive technology for blind and low-vision users, where a scene-description system that reports reflected objects as real could send a user toward a wall.
• Augmented and virtual reality, where correct separation of real and virtual geometry underpins placement and occlusion.
• Warehouse and industrial robotics operating near glass partitions, screens, and polished floors.

Industry relevance. Any product that fuses a depth camera with a language model — navigation stacks, smart-home assistants, inspection drones, service robots — inherits this failure mode. The finding that adding depth can lower accuracy is directly actionable for teams currently treating depth as a free accuracy boost.

Future Directions

• Dynamic and video scenes. The benchmark is built on static images, but embodied systems must reason over changing viewpoints and temporal cues. The authors name visual navigation as a motivating case.
• Alternative range sensing. Depth here comes from RGB-D cameras; the authors suggest LiDAR or sonar could change the reliability profile and warrant separate evaluation.
• Broader coverage of reflection physics. The authors acknowledge it is infeasible to exhaustively cover all reflective materials, lighting conditions, mirror geometries, and multi-mirror configurations.
• Reflection-aware model design. Since mirror masks gave only a modest gain and depth actively hurt, an open question is what architectural or training change would let a model discount corrupted depth rather than trust it. The paper diagnoses the failure but does not propose a fix.

Target Audience

Researchers and engineers working on multimodal LLMs, embodied AI, and robot perception — particularly those building RGB-D pipelines who currently assume depth is reliable. It is also valuable for benchmark designers interested in how task hierarchy and referring-protocol controls should be structured, and for practitioners deploying vision-language systems in indoor environments where mirrors, glass, and polished surfaces are unavoidable. Readers need working knowledge of VLMs and depth sensing; the paper is not an introductory survey.

Authors’ abstract

Vision-language models (VLMs) have made significant progress in spatial reasoning, particularly by integrating RGB images with depth data, which is often assumed to faithfully reflect physical distances. However, mirrors violate this assumption: they distort depth readings while reproducing object appearances, making reflections difficult to distinguish from real objects. As existing benchmarks rarely consider such reflective scenarios, VLM performance in mirror scenes remains largely unexplored. In this work, we introduce a mirror-centric RGB-D dataset from reflective scenes to evaluate the spatial reasoning capabilities of state-of-the-art VLMs in these scenarios. Specifically, we collect 1,011 RGB-D images from 525 real-world scenes, spanning diverse indoor and outdoor environments, six reflective surface types, and both daytime and nighttime conditions. All images are annotated with reflective surface region masks and mask annotations for both directly observed and mirror-reflected objects. Then, we curate a benchmark named MIRA-Bench (Mirror-Induced Reasoning Ambiguity Benchmark), comprising 14,140 QA pairs across eight sub-tasks organized into three cognitive levels: Reflection-aware Perception, Spatial Reasoning, and Scene-level Decision-making. We evaluate state-of-the-art VLMs on MIRA-Bench and find consistent failures across tasks: models misclassify whether objects are directly observed or reflected, fail to match reflected instances to their real-world counterparts, and produce erroneous spatial inferences. Remarkably, providing depth data actually degrades reasoning performance, implying that VLMs cannot identify incorrect depth in mirror scenes. The code and data are available at https://github.com/EvaYin-cloud/MIRA-Bench