SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

AI summary

Overview

Research area: Computer vision and multimodal evaluation — specifically image quality assessment (IQA) extended to scientific figures embedded in full manuscripts.

Technical level: Intermediate. The core idea is intuitive, but the evaluation machinery (gated multi-dimensional rubrics, MAE/SRCC/Krippendorff's alpha, staged agent protocols) assumes some familiarity with benchmark design and vision-language model evaluation.

Scope: The paper introduces a 7,609-figure benchmark drawn from published ACL, EMNLP, ICML, and NeurIPS papers, plus a staged agent that scores each figure against its caption and its citing paragraphs on five quality dimensions.

What This Paper Is About

Existing image quality assessment methods judge photographs or AI-generated images from pixels alone — color, sharpness, aesthetics, text-image similarity. Scientific figures are different: a truncated axis, a hidden baseline, or a caption that contradicts the plot is invisible to pixel-level inspection but obvious once you read the manuscript. This paper builds a benchmark and an evaluation protocol that score published figures only when they are bound to their caption and the paragraphs in the paper that cite them, testing whether a model can assess a figure in the way a peer reviewer would.

Key Contributions

1. SciFigQual-Bench: A full-manuscript-context benchmark of 7,609 curated figures from 1,144 papers (6,308 expert-rated), crawled from 62,694 PDFs across ACL, EMNLP, ICML, and NeurIPS (2020–2025), with roughly 355,000 index-resolved citing-paragraph records. Each instance is a tuple of figure crop, caption, citing paragraphs, and metadata.

2. A five-dimensional gated rubric: Visual Clarity (VC), Structure & Layout (SL), Caption Consistency (CC), Context Consistency (CTX), and Misleading Risk (MR), each scored on a 1–10 integer scale. The L1 gating rule hides dimensions when their required evidence is absent, so missing captions or citations are never penalized as low quality.

3. SFQ-Agent: A staged cross-modal judge that collects vision evidence and language evidence separately, fuses them in a dedicated judge stage, and passes the result through a deterministic "Runner" that locks visual scores to their originating module and applies rule-based caps. This is contrasted against single-pass Direct and OCR-augmented Sidecar protocols.

4. eval1200: A frozen 1,200-figure public test split (254 papers) with a 29-configuration protocol–backend ablation, plus a separate human-agreement ceiling analysis that clarifies what model scores do and do not mean.

Main Findings

• Staged judging wins, but narrowly: SFQ-Agent with GPT-5.6-Sol (F3) achieves the best overall result — MAE 0.418, Within-1 93.4%, SRCC 0.598, bias +0.012 — versus Direct-best (Claude-Opus-4.8, MAE 0.443) and Sidecar-best (GPT-5.6-Sol, MAE 0.436). The Direct-best to Agent-best gap is only 0.025 MAE, at triple the call budget.

• The constant-mean baseline is a strong floor: Predicting the mean score (8.13) for every figure already yields MAE 0.565 and Within-1 87.9%. Within-1 therefore separates protocols poorly; MAE and SRCC carry more signal.

• Human agreement is far lower than model-gold agreement: Human pairwise Within-1 is 71.5% overall (Krippendorff α = 0.51), with leave-one-out MAE 0.70. The Agent's 93.4% is measured against mean gold, not against an individual rater, so it cannot be read as superhuman caption verification.

• Caption Consistency is the unstable construct and the typical defect: CC has the lowest human agreement (pairwise Within-1 52.5%, α = 0.49, LOO MAE 1.50) and the lowest mean score in the dataset (7.42 versus SL at 8.58). Caption–figure mismatches, not blur, are the dominant quality problem in published figures.

• Citing text does not make caption agreement easier: Human CC Within-1 is lower with citing text present (50.0%, n = 807) than without (59.5%, n = 393), though the split is observational.

• OCR side features help every backend tested: Adding PaddleOCR-VL and classical CV descriptors in the Sidecar protocol reduces MAE across all nine backends where it was run, with the largest drop on GLM-4.6V (ΔMAE = 0.044).

• Backend behavior diverges meaningfully: Qwen-VL-Max is calibrated under Direct (Within-1 88.8%) but ranks poorly (SRCC 0.416) because of high CC/CTX error. Doubao-Seed-2.0-pro is systematically harsh (bias −0.120). GLM-4.6V benefits most from richer protocols but remains weakest overall. GPT-5.6-Sol is the most balanced.

• Scores are compressed: Published-only figures cluster near 8, making fine-grained ranking harder than coarse Within-1 suggests. Even the best SRCC is 0.598.

• Scale exceeds prior benchmarks: 7,609 released figures (6,308 rated) versus SIQA-U (2,240), SPUR (1,084), VisJudge (3,090), SCIEval (6,000), and GENFIG1 (584), with uniquely full PDF-and-section binding and index-driven citing paragraphs.

Methodology in Plain English

The authors built the benchmark in stages. First, they crawled tens of thousands of PDFs from OpenReview, ACL Anthology, and PMLR, filtered to 2020–2025 and four venues, and de-duplicated by content fingerprint. Next, they rasterized each PDF at 300 DPI and used layout parsing (Marker) to find figure bounding boxes and captions, then ran a five-stage curation pipeline that hard-deletes junk crops, flags anomalies without deleting them, re-crops broken panels, attaches soft quality metadata, and finally drops whole papers whose figures are mostly unusable.

For context binding, they scanned the body text of each PDF with PyMuPDF for figure-index patterns ("Figure 3", "Fig. 3"), collected the hosting paragraphs from the same section, merged multi-sentence windows, and capped token length. This produced roughly 355,000 citing-paragraph records tied to specific figures by index — not by heuristics over abstracts or introductions.

Human annotation used a calibrated rubric. A 50-figure pilot aligned annotators, every score had to be justified with a specific visible mark, caption phrase, or citing sentence, and a dual-annotated ACL 2025 holdout triggered adjudication when raters differed by more than two points. Annotations were aggregated dimension-first rather than by averaging per-rater overalls.

The SFQ-Agent mirrors how a careful reviewer works. It extracts visual facts (OCR text regions, blur, edge density, color histograms, axis units, legend notes) in one pass. In a separate pass, a language model reads only the caption, citing paragraphs, and metadata — never the pixels — and extracts textual claims and rule violations. A third stage fuses the two evidence reports to score CC, CTX, and MR, including conflict detection (for example, an upward trend in the figure described as "degradation" in the text). A deterministic post-processor then locks VC and SL to the vision module, applies caps for bare-pointer citations and vague captions, and averages the surviving dimensions under the gating rule. The three protocols share identical inputs and rubric text; they differ only in how many calls they make and whether evidence is staged.

Why This Matters

Impact on research: The paper reframes scientific figure evaluation as a tri-modal problem — image, caption, and citing prose — and provides both a benchmark and a reproducible protocol for it. It also supplies an honest ceiling analysis showing that CC is a genuinely unstable human construct, which explains why model scores against mean gold look deceptively strong. This distinction between "fit to the mean" and "agreement with a rater" is a methodological contribution that applies well beyond figure assessment.

Real-world applications:

• Pre-submission figure checking: authors scanning manuscripts for truncated axes, missing baselines, vague captions, and text-figure contradictions before peer review.
• Journal and conference screening: editorial triage of large submission volumes for figures that need human attention.
• Automated peer-review assistance: surfacing specific, evidence-cited concerns rather than opaque quality scores.
• Reproducibility auditing: flagging figures whose captions or cited claims do not match what the plot actually shows.

Industry relevance: Publishers, preprint servers, and conference program committees handle figure QA at a scale no human reviewer can match. The benchmark's auditable scoring rationale — where each score traces back to structured evidence — is designed for exactly that setting, and the finding that caption verification is the bottleneck tells tool builders where to invest rather than chasing incremental visual-quality gains.

Future Directions

1. Closing the caption verification gap: CC remains the weakest axis for both humans and models. A dedicated, trained caption-figure verification module — rather than a prompted foundation model — is the obvious next step.

2. Extending beyond accepted papers: The pool contains only published CS figures, so rejected-manuscript defects are unobserved and scores compress near 8. Including preprints, withdrawn papers, or author drafts would widen the score distribution and test ranking more meaningfully.

3. Equalizing compute and cost: The 0.025 MAE gap between Direct and Agent is not corrected for the threefold call budget, and no confidence intervals are reported. A cost-matched comparison would clarify whether staging genuinely pays off.

4. Handling LLM and backend drift: The reported rankings depend on frozen run IDs of closed-source models, which change over time. A periodically refreshed or open-weight-anchored leaderboard would make the benchmark durable.

Target Audience

Benchmark designers and multimodal evaluation researchers will find the gating scheme and the human-ceiling analysis directly reusable. Vision-language model developers get a stress test that single-pass judges demonstrably fail. Peer-review tooling and academic publishing engineers get a concrete task definition and a released dataset with a project page and code. Scientific writing instructors and journal editors can use the five-dimensional rubric as a checklist independent of any model. Readers new to IQA should start with the introductions and Section 3.1; the experimental tables demand prior familiarity with agreement metrics.

Authors’ abstract

Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers. However, existing image quality assessment (IQA) methods are predominantly designed for natural photographs or AI-generated content, which cannot be directly applied to scientific papers. The few existing studies on scholarly charts remain confined to visual-surface comparisons, failing to verify caption alignment, citation relevance, or visual misleadingness. To address this, we propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions (clarity, layout, caption fit, context relevance, and misleading risk). The data covers top computer-science conferences from 2020 to 2025; 6,308 images were independently scored by multiple domain experts in five dimensions and aggregated into gold-standard annotations. Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context. To enable automated evaluation on this benchmark, we designed a staged cross-modal evaluation framework SFQ-Agent to achieve auditable and refined scoring through the collection and fusion of modal evidence. Multiple mainstream large models were evaluated on the test subset eval1200, and SFQ-Agent (F3) equipped with GPT-5.6-Sol achieved the lowest overall average absolute error (0.418) and the highest consistency rate (93.4%), consistently outperforming both direct evaluation and auxiliary (Sidecar) visual language model evaluation schemes.