PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies

AI summary

Overview

Research area: Computer vision and multimodal large language model (LMM) evaluation, specifically benchmarks for scientific document understanding and cross-modal consistency checking.

Technical level: Intermediate. The paper assumes familiarity with multiple-choice benchmarking, multimodal prompting, and chain-of-thought evaluation, but its framing problem (papers containing mismatched figures, tables, text, and equations) is accessible without specialist background.

Scope: The paper introduces PRISMM-Bench, a benchmark of 384 real, reviewer-flagged inconsistencies drawn from 353 ICLR 2024 and 2025 submissions, and uses it to evaluate 21 LMMs on three tasks (identification, remedy, and pair matching) under three levels of input context.

What This Paper Is About

Scientific papers frequently contain subtle inconsistencies between their text, figures, tables, and equations, such as a reward function defined one way in a figure and differently in the prose. These errors are hard to detect, require domain knowledge, and undermine clarity, reproducibility, and trust.

Existing benchmarks either isolate single modalities (diagrams, charts, or tables alone) or rely on synthetic, artificially injected errors that do not reflect real-world complexity. PRISMM-Bench addresses this gap by mining authentic inconsistencies that human reviewers explicitly flagged during peer review, then converting them into a tiered set of multimodal question-answering tasks.

Key Contributions

1. A reviewer-sourced dataset of real multimodal inconsistencies. The authors curate 384 validated inconsistencies across 353 ICLR 2024 and 2025 submissions, spanning 15 categories of visual-textual and inter-visual mismatches, grounded in actual peer-review comments rather than synthetic perturbations.

2. A three-task benchmark suite. The benchmark decomposes inconsistency handling into Inconsistency Identification (detect what the inconsistency is), Inconsistency Remedy (determine how to fix it), and Inconsistency Pair Match (identify which two elements conflict), evaluated across Focused, Page, and Document context settings.

3. A JSON-based debiasing method for multiple-choice evaluation. The authors are the first to propose converting free-form natural-language answer options into uniform structured representations (an Evidence–Claim JSON format for Identification and a Target–Action JSON format for Remedy) to suppress choice-only shortcuts and linguistic biases.

4. A 21-model evaluation. Sixteen open-weight models of varying scale, two specialized high-resolution models, and three proprietary models are benchmarked, exposing current limitations in detecting, understanding, and correcting inconsistencies in scientific papers.

Main Findings

• Overall performance is strikingly low. Across all evaluated models, accuracy ranges from 27.8% to 53.9%. The authors characterize this as underscoring the difficulty of multimodal scientific reasoning.

• Proprietary models lead but remain far from reliable. GPT-5 with high reasoning reaches the highest average of 53.9%, followed by Gemini 2.5 Pro at 52.8% and GPT-5 (minimal) at 44.0%. The best open-weight model, GLM-4.5V 106B, reaches 42.5% in the text (listed as 42.6 in Table 1), matching GPT-5 (minimal reasoning) but trailing the high-reasoning variant by 11.4 points.

• Expanding context hurts performance. Models do best under Focused Context and often degrade toward random chance under Page and Document inputs, which the authors attribute to distraction and long-range grounding difficulties in dense, multi-page inputs.

• Remedy is harder than detection. Remedy scores are consistently lower than Identification scores across models, indicating that proposing corrections requires deeper reasoning than merely spotting a mismatch.

• Scale does not guarantee accuracy. Gemma 3 12B achieves 63.5% on Match, rivaling proprietary models, while InternVL3 78B lags at 45.3%. The authors conclude that architectural design, not just scale, is critical for relational reasoning.

• Reasoning helps substantially. InternVL3.5 8B outperforms its non-reasoning predecessor InternVL3 8B by over 5 percentage points, achieving accuracy comparable to models with nearly nine times more parameters. Disabling chain-of-thought reduces accuracy by up to 14 points (16.6% relative for GLM-4.5V, 18.0% for InternVL3.5 8B, and 25.7% relative / 14 points absolute for InternVL3.5 38B).

• Natural-language multiple-choice answers inflate apparent model ability. Without any context, Gemini 2.5 Pro reaches 70.1% on natural-language options versus 37.3% on JSON options; on the full Identification dataset it reaches 61.2% without context in natural language versus 37.8% in JSON. Human participants score 27.5% without context, near chance, showing they cannot exploit the same linguistic regularities.

• Humans rely more on visual evidence than models do. Using the Visual Reliance Ratio R adapted from the normalized Perceptual Score, human participants achieve 69.0% while the top model, InternVL3.5 8B, achieves 53.5%.

• JSON formatting raises visual reliance across models. On the full Identification set, R increases under JSON for every model tested: InternVL3.5 8B from 17.1 to 29.3, InternVL3.5 38B from 22.5 to 38.1, Qwen 2.5 VL 72B from 16.1 to 20.8, and Gemini 2.5 Pro from 43.8 to 45.2.

• Resolution ablation confirms the 144 DPI baseline. Lowering to 72 DPI hurts most models, with drops up to 20.1% relative for InternVL3.5 38B and 3.3% for Gemini 2.5 Pro, though VILA HD 4K 8B showed a slight gain at 72 DPI. Raising to 300 DPI does not consistently help.

• Reasoning models struggle with structured inputs. Without context, Model accuracy on JSON falls toward human levels (InternVL3.5 38B drops from 53.7% to 25.3%), and with context plus JSON, LMMs no longer match human natural-language performance.

Methodology in Plain English

The authors built the benchmark in six stages.

First, they sourced reviews from ICLR 2024 and 2025 submissions using the OpenReview API v2, restricting to rejected or withdrawn submissions without rebuttals, on the reasoning that issues resolved during rebuttal would not persist in final PDFs. This yielded 18,009 reviews.

Second, because manual screening of all reviews was infeasible, they used Mistral Nemo 2407 with low temperature settings to summarize reviews and flag potential inconsistency mentions, producing 6,056 candidate inconsistencies across 2,458 reviews.

Third, human annotators used a custom web-based tool that displayed one flagged comment alongside the paper in an embedded PDF viewer. Annotators confirmed whether the comment described a factual, identifiable inconsistency, then marked the relevant text (page, line, content) and cropped the relevant visual regions, assigning each case a category and description. This produced 384 validated inconsistencies across 353 submissions.

Fourth, they generated multiple-choice tasks with four options each (one correct, three distractors), with distractors generated using Gemini 2.5 Flash and manually refined. Identification asks what the inconsistency is; Remedy asks what action resolves it; Match applies only to the 192 samples involving two distinct visual elements and asks the model to pick the counterpart of a given element.

Fifth, they introduced structured JSON answer representations after pilot experiments showed models scored well above chance even without any visual context (Gemini 2.5 Flash reached 57.6% on Identification without context against 25% random chance). Text manipulation of distractors proved insufficient, so answers were converted into Evidence–Claim and Target–Action JSON formats, reducing Gemini 2.5 Flash's no-context accuracy to 34.0%. A 20% subset was manually verified for semantic fidelity.

Finally, they evaluated models under three context settings: Focused (only the minimal cropped visual and precise text passage), Page (a 144 DPI rasterized full page image), and Document (the whole paper, segmented into 5 collage images each holding n_pages/5 pages in a 3-column grid, following MMLongBench-Doc; LLaVA Onevision 7B and 72B used 3 images at n_pages/3 to fit context limits). All models were run with greedy decoding via vLLM v0.10.1 on 4× NVIDIA A100 64GB GPUs, except InternVL3.5 (8B, 38B) which needed temperature 0.6 and GPT-5 (minimal, high) which has a fixed temperature of 1.0.

Why This Matters

Impact on research. The paper reframes scientific document understanding as a consistency-checking problem rather than a question-answering problem, and it supplies evidence that common multiple-choice evaluation inflates LMM scores through linguistic shortcuts. The measured 17.0% rate of ICLR 2025 submissions containing at least one reviewer-flagged inconsistency gives the problem a concrete scale within one major venue.

Real-world applications:

• Automated pre-submission or pre-publication screening that flags mismatches between a paper's claims, figures, tables, and equations.
• Reviewer assistance tools that surface candidate inconsistencies for human experts to confirm, reducing manual cross-checking burden.
• Reproducibility auditing of published research, where inconsistent notation or outdated reported results can mislead replication attempts.
• Multimodal document processing in technical domains where text, charts, and formulas must agree, such as engineering specifications and financial or medical reports.

Industry relevance. The finding that JSON-structured answers suppress shortcut exploitation is directly useful to anyone building evaluation harnesses, since it suggests that natural-language multiple-choice benchmarking systematically overstates model capability. The result that reasoning capability contributes more than raw parameter count to this task also informs model selection and architecture decisions for document-analysis products.

Future Directions

• Expand domain coverage. The authors state the benchmark is currently limited to AI-domain papers from ICLR 2024 and 2025 and emphasizes rejected submissions; they call for expansion to other fields and venues.
• Include accepted papers. Extending to inconsistencies that persist in accepted papers would produce a broader and more representative testbed.
• Improve the filtering pipeline. The authors list improvements in the filtering pipeline as future work, implying the current LLM-assisted review mining step has room for greater recall or precision.
• Develop stronger debiasing strategies for long-form multiple-choice evaluation. The JSON approach is presented as a first step, and the paper motivates continued work on removing linguistic shortcuts in benchmark design.
• Advance reasoning architectures. The authors argue that future progress requires reasoning architectures that move beyond error detection toward correction, plus more robust grounding over long, distractive contexts.

Target Audience

This paper is most useful to researchers building or auditing multimodal benchmarks, particularly those working on scientific document understanding, document-level visual question answering, and evaluation methodology for multiple-choice formats. It is also valuable to LMM developers interested in long-context grounding and chain-of-thought reasoning, and to practitioners designing automated review, reproducibility-checking, or technical document validation systems that must reason across text, figures, tables, and equations. Readers interested in peer review analytics and scientific integrity will find the reviewer-sourced dataset design and the 17.0% flagged-inconsistency rate relevant as well.

Authors’ abstract

Large Multimodal Models (LMMs) are increasingly applied to scientific research, yet it remains unclear whether they can reliably understand and reason over the multimodal complexity of papers. A central challenge lies in detecting and resolving inconsistencies across text, figures, tables, and equations, issues that are often subtle, domain-specific, and ultimately undermine clarity, reproducibility, and trust. Existing benchmarks overlook this issue, either isolating single modalities or relying on synthetic errors that fail to capture real-world complexity. We introduce PRISMM-Bench (Peer-Review-sourced Inconsistency Set for Multimodal Models), the first benchmark grounded in real reviewer-flagged inconsistencies in scientific papers. Through a multi-stage pipeline of review mining, LLM-assisted filtering and human verification, we curate 384 inconsistencies from 353 papers. Based on this set, we design three tasks, namely inconsistency identification, remedy and pair matching, which assess a model's capacity to detect, correct, and reason over inconsistencies across different modalities. Furthermore, to address the notorious problem of choice-only shortcuts in multiple-choice evaluation, where models exploit answer patterns without truly understanding the question, we further introduce structured JSON-based answer representations that minimize linguistic biases by reducing reliance on superficial stylistic cues. We benchmark 21 leading LMMs, including large open-weight models (GLM-4.5V 106B, InternVL3 78B) and proprietary models (Gemini 2.5 Pro, GPT-5 with high reasoning). Results reveal strikingly low performance (27.8-53.9\%), underscoring the challenge of multimodal scientific reasoning and motivating progress towards trustworthy scientific assistants.