OneScene-Bench: Do Vision-Language Models Integrate Evidence Across Views?

AI summary

Overview

Research area: Computer vision and multimodal evaluation — specifically, whether vision-language models (VLMs) actually fuse evidence from multiple photographs of the same physical scene.

Technical level: Intermediate. The paper is readable without deep math, but it assumes familiarity with vision-language models, multi-image prompting, and benchmark design.

Scope: The authors build and run a 1,300-question, human-verified benchmark over real-world two-view captures to test whether twelve VLMs integrate evidence across viewpoints rather than processing each image independently.

What This Paper Is About

VLMs increasingly accept several images at once, but answering correctly does not prove the model combined information from those images — many multi-view questions can be solved by counting or judging each image separately and applying a simple rule. The authors build OneScene-Bench to separate questions that are solvable per-view from questions that genuinely require linking objects across viewpoints. The goal is to measure the gap between processing multiple images and integrating them into one coherent scene representation.

Key Contributions

1. OneScene-Bench: a controlled benchmark of 1,300 manually verified examples built from real-world multi-view captures in DL3DV-10K, covering multi-view counting (700 questions), cross-view consistency (300), and multi-view spatial reasoning (300), with two views per example.

2. An overlap-based counting taxonomy: counting examples are split into complete overlap (200 questions), subset overlap (300), and complementary views (200). The first two admit a "take the larger per-view count" shortcut; only complementary views force the model to identify which instances are shared and deduplicate them.

3. A visibility-controlled consistency intervention: localized objects or artifacts are inserted with gpt-image-1.5 either into regions observable from both viewpoints (creating a genuine contradiction, labeled inconsistent) or into view-specific regions (leaving the pair compatible, labeled consistent). Both conditions contain an image-level difference, so difference detection alone cannot separate them.

4. A broad diagnostic evaluation: twelve open-weight and proprietary VLMs under an open-ended protocol with a GPT-4o judge, plus analyses of per-view aggregation heuristics, image-order sensitivity, no-image bias, instruction-tuned versus reasoning-enabled variants, and scaling from 2 to 6 input views.

Main Findings

• Counting collapses on complementary views. Accuracy across the twelve models ranges from 45.5–81.5% on complete overlap and 26.7–76.7% on subset overlap, but only 4.5–50.0% on complementary views, where cross-view correspondence and deduplication are required.

• Even the strongest models show the drop. Gemini 3.7 Flash falls from 81.5% (complete overlap) to 50.0% (complementary); GPT-5.5 from 70.0% to 38.5%; Qwen3-VL-32B-Instruct from 70.0% and 70.7% to 34.0%; InternVL3.5-38B-HF from 72.0% to 14.5%. Phi-4-multimodal-instruct is lowest on complementary views at 4.5%.

• Cross-view consistency sits at chance. Accuracy ranges from 47.0% to 55.7% across all models. LLaVA-OneVision-2-8B-Instruct and Phi-4-multimodal-instruct answer "yes" for every example and therefore score exactly 50.0% on the balanced set, indicating a default assumption that different views agree rather than actual verification.

• Spatial reasoning is markedly stronger. GPT-5.5 reaches 92.7% and Gemini 3.7 Flash 89.7%. Kimi-VL-A3B-Instruct is an outlier at 15.7%, frequently naming object B instead of the required reference object A or C, which the authors attribute to failure to follow the answer format.

• Joint answers look like max-aggregation, not summation. Comparing joint predictions against single-view predictions, agreement with max(C1, C2) consistently exceeds agreement with C1 + C2 — including on complementary views where the maximum is provably insufficient. For example, LLaVA-OneVision-2-8B-Instruct reaches 62.0% max agreement versus 4.5% sum agreement on complementary views.

• Explicit reasoning usually hurts counting. Reasoning-enabled variants score lower almost everywhere: Qwen3-VL-32B-Thinking reaches 59.0/57.0/26.5 versus 70.0/70.7/34.0 for the instruct variant; MiniCPM-V-4.5 (Thinking) reaches 61.5/44.3/9.5 versus 67.0/51.7/20.5. The only exception is a 1.5-point gain for Kimi-VL on complementary views.

• More views help only up to a point. On complete-overlap counting with N = 2 to 6 views, accuracy initially improves but peaks before N = 6 for all four models tested, so additional observations do not yield monotonic gains.

• Models retain non-trivial accuracy without any images. Removing the images while keeping the questions leaves substantial residual accuracy (for example, Qwen3-VL-32B-Instruct at 37.5% on complete overlap versus 70.0% with images), revealing language and answer-distribution priors. Gemma-4-31B-IT refuses all 300 consistency questions and 293 of 300 spatial questions in this setting, while Qwen3-VL-32B-Instruct, InternVL3.5-38B-HF, and LLaVA-OneVision-2-8B-Instruct refuse none.

• Image order is not a major confound. Across the four open-weight models tested, 18 of 20 model–task combinations change by at most four percentage points when the two views are swapped; the largest changes are 6.0 points for Gemma-4 on complete-overlap counting and 9.0 points for LLaVA-OneVision-2-8B-Instruct on spatial reasoning.

Methodology in Plain English

The authors start from DL3DV-10K, a dataset of real-world scenes filmed from many viewpoints. They sample two frames from the same location with a frame-index gap of 10 to 25, which gives moderate viewpoint change while keeping visual overlap.

For counting, a VLM (Qwen3-VL-32B-Instruct) proposes an object category that is clearly visible and countable in both frames and estimates per-view and unique-instance counts. The overlap type (complete, subset, or complementary) is then assigned by hand after checking the visible instance sets.

For consistency, they match local features between the two views with SIFT, filter geometrically inconsistent matches with RANSAC, and use the density of verified matches to identify regions shared by both viewpoints versus regions visible from only one. An object or artifact is then inserted into one of these regions with gpt-image-1.5 and blended in with a Gaussian-smoothed mask. An edit in a view-specific region is labeled consistent; an edit in a shared region is labeled inconsistent.

For spatial reasoning, the same VLM proposes three objects A, B, C spread across the two views such that comparing whether B is closer to A or to C requires combining evidence from both images.

All generated or proposed candidates pass through manual review — human annotators check object identity, visibility, cross-view correspondence, overlap type, and the final label, and discard ambiguous cases. Evaluation is open-ended: models write free-form responses, and GPT-4o judges whether the response reaches the correct conclusion, accepting semantically equivalent formats.

Why This Matters

Impact on research. Prior multi-image and multi-view benchmarks report strong scores but do not control whether cross-view integration is actually needed, so a correct answer may come from independent per-view processing. This paper shows that controlling the required evidence changes the picture dramatically: models that look competent on complete-overlap counting fall to between 4.5% and 50.0% when correspondence is unavoidable, while spatial reasoning stays above 70% for most models. The benchmark therefore argues that future multi-view evaluation must be diagnostic rather than merely difficult.

Real-world applications:

• Robotics and embodied agents: an agent that moves around a room and must decide how many distinct objects it has seen, rather than double-counting the same chair from two angles.
• Autonomous driving and inspection: reconciling camera feeds from different positions to decide whether a scene has genuinely changed or an object is simply out of frame.
• 3D scene understanding from photographs: building object-centric scene representations from sets of images, where deduplication across views is a prerequisite.
• Surveillance and change detection: distinguishing a real scene-level contradiction from a difference caused by occlusion or viewpoint.

Industry relevance. Multi-image input is now a standard capability in commercial and open-weight VLM APIs, and these models are increasingly positioned as interfaces to 3D environments. The finding that consistency accuracy stays near 50% — including for recent proprietary models — is a concrete reliability warning for products that assume a model checks whether two views agree rather than defaulting to "yes." The benchmark and code are released at https://github.com/swathisuhas/OneScene-Bench.

Future Directions

• Broadening the benchmark. The authors explicitly note that OneScene-Bench covers only three capabilities and 1,300 verified examples from a single source dataset (DL3DV-10K); expanding across tasks, environments, and models is left to future work.

• Closing the correspondence and deduplication gap. The paper identifies instance-level cross-view correspondence as the specific bottleneck, and notes that strong spatial reasoning does not transfer to it — a targeted capability to build and measure.

• Making consistency judgments visibility-aware. Models respond similarly to edits in shared and view-specific regions, so a model that explicitly reasons about what should be observable from another viewpoint is an open design goal.

• Understanding why additional views stop helping. Accuracy peaks before six views for every model tested; diagnosing whether this comes from identity confusion, double-counting, or context limits is unresolved.

Target Audience

Researchers and engineers working on multimodal LLMs, multi-view and 3D scene understanding, and benchmark design. It is most useful for those who build evaluation suites for vision-language models, for roboticists and embodied-AI practitioners who rely on multi-camera input, and for product teams deciding whether current VLM multi-image support is trustworthy for scene-level reasoning. The paper is intermediate in difficulty: accessible to graduate students and practitioners familiar with VLMs, with the technical weight in the benchmark design rather than in formal methods.

Authors’ abstract

Vision-language models (VLMs) increasingly process multiple views of the same scene, but it remains unclear whether they integrate these observations into a coherent scene-level representation. Existing multi-image evaluations do not isolate this capability, as tasks may admit correct solutions through independent per-view processing without requiring cross-view correspondence. We introduce OneScene-Bench, a controlled benchmark designed to explicitly distinguish when cross-view integration is necessary. Our benchmark evaluates three complementary capabilities on real-world multi-view scenes: unique object counting, cross-view consistency, and spatial reasoning. For counting, we control object overlap across views to separate cases solvable by simple per-view aggregation from those requiring instance-level correspondence. For consistency, we introduce localized edits in shared and view-specific regions, testing whether models distinguish genuine scene contradictions from differences caused by viewpoint and visibility. For spatial reasoning, we construct object relations that require combining spatial evidence across viewpoints to compare physical proximity. We evaluate twelve state-of-the-art open-weight and proprietary VLMs and find that counting accuracy drops substantially on complementary views, ranging from 4.5-50.0%, compared with 45.5-81.5% on complete-overlap examples. Moreover, cross-view consistency remains around 50% across models, including recent proprietary VLMs, despite substantially stronger performance on spatial reasoning. Our results reveal a substantial gap between processing multiple images and integrating evidence across them, highlighting the need for evaluations that explicitly test scene-level multi-view understanding. Code and data are available at https://github.com/swathisuhas/OneScene-Bench.