RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension

AI summary

Overview

Research area: Natural Language Processing — evaluation benchmarks for document and scholarly-paper understanding by large foundation models.

Technical level: Intermediate. Readers should be comfortable with LLM benchmarking, retrieval-augmented generation, and standard QA evaluation metrics, but the paper is readable without deep mathematical background.

Scope: This paper introduces RPC-Bench, a large-scale, peer-review-grounded question-answering benchmark that measures how well 28 state-of-the-art language, vision-language, document-centric, and RAG models actually comprehend computer science research papers.

What This Paper Is About

Foundation models are increasingly used as research assistants, yet there is no rigorous way to tell how well they truly understand a scientific paper rather than just retrieving surface text. Existing benchmarks are either small, built from synthetic questions, or organized by shallow task types that don't capture the depth of understanding a paper actually demands. RPC-Bench addresses this by harvesting real questions raised by peer reviewers during review–rebuttal exchanges on OpenReview and turning them into a large, fine-grained, human-verified QA benchmark organized around the natural research workflow.

Key Contributions

1. A large-scale, authentic benchmark. RPC-Bench contains 4,150 computer science papers (2013–2024) and 61.3K QA pairs, of which 15K are human-verified. Questions derive from genuine reviewer questions and author rebuttals rather than synthetic generation, so every answer is grounded in the published paper.

2. A workflow-aligned, fine-grained taxonomy. Questions are organized into 4 primary dimensions — Concepts, Methods, Experiments, and Claim Verification — expanded into 9 categories and mapped to what, how, and why question types, reflecting the natural progression of scientific reasoning rather than generic task labels.

3. An LLM–human collaborative annotation framework. A pipeline using GPT-4o to segment reviews into comment–response units, GLM-4-Plus and DeepSeek-V3 to rewrite them into QA pairs, plus automated filtering, then multi-round human annotation and review with measured inter-annotator agreement.

4. A multi-dimensional evaluation protocol. Instead of single accuracy scores, the framework scores open-ended answers on correctness, completeness, and conciseness using multiple LLM judges, deriving an F1-like score and an informativeness score, and validating judge alignment against human pairwise preferences.

Main Findings

• Even the strongest model falls far short. GPT-5 achieves only 68.2% F1-like (the harmonic mean of correctness and completeness), and this drops to 37.46% informativeness once conciseness is factored in. No model comes close to reliable research-paper comprehension.

• Traditional metrics are misleading. ROUGE-L and BERTScore cannot distinguish large models from small ones. Monkey(V), a small document-centric model, posts the best ROUGE-L (20.16%) while its correctness and completeness are only 17.08% and 11.27%.

• Text beats images for the same model. For multimodal-capable models, swapping text for rendered page images reduced F1-like by 4.74–36.1% (Qwen3 fell from 56.26% to 20.16%). The largest losses appear in correctness and completeness, indicating current VLMs cannot yet exploit scholarly figures and tables effectively.

• Small document-centric models fail structurally. Models around 8B parameters score only 8–18% F1-like and frequently produce repetitive or incoherent output, showing general-domain fine-tuning is not enough for long, structured academic documents.

• Deeper question types are harder. Models perform best on concept understanding and method disambiguation, and worst on tasks requiring reasoning about experimental analysis and results — the gap widens further with image inputs.

• Claim verification remains weak and instruction-following is fragile. Accuracy on true/false claim verification is limited, and some models (e.g., Claude-4, HippoRAG2) frequently fail to emit the required "true"/"false" format at all.

• Concise answers are the easiest thing to learn. Fine-tuning Qwen and LLaMA on the training split improved informativeness by 11.38% and 10.64%, but almost entirely through better conciseness while F1-like stayed flat — correctness and completeness are the hard parts.

• Judge design matters. Including title and abstract, evaluating each dimension separately rather than jointly, and using decimal (not integer) scoring all improved agreement with human judgment. The best judge configuration (GPT-5 with decimal scoring) reached 0.8535 average agreement.

Methodology in Plain English

The researchers started from OpenReview, crawling 44.7K peer-reviewed papers along with their public review threads and author rebuttals. They cross-checked against the AMiner academic search system to remove incomplete entries, leaving 17.7K papers. To avoid bias toward only successful work, they sampled 3,521 accepted papers with at least 50 citations, plus 361 highly cited rejected papers and 361 randomly chosen rejected papers, for a final collection of 4,243 papers.

Because review threads contain long, messy prose rather than clean question–answer pairs, they used GPT-4o to split each review into minimal self-contained comment–response units. Then GLM-4-Plus and DeepSeek-V3 rewrote each unit into either a free-form QA pair or a true/false claim verification item, and assigned it to a taxonomy category. A filtering pass using GLM-4-Plus discarded items that couldn't be answered from the paper alone — editorial nitpicks, questions depending on external resources, or vague promises without a substantive answer.

Human annotators with master's degrees or higher then verified the output, going through training until reaching a 95% pass rate and capping production at 80 QA pairs per person per day. Reviewers independently checked the annotations and removed an additional 8.87% of items judged low-quality or unanswerable from the final paper. They measured labeling agreement with Cohen's Kappa: 0.72 for category assignment among annotators and 0.78 among reviewers, with 0.81 and 0.85 for question retention — indicating strong consensus.

For evaluation, papers were converted to Markdown text with MinerU, or rendered as page images with PyMuPDF at 200 DPI (first 15 pages). Models answered under minimal constraints: rely only on the given paper, be concise and under 3,000 characters for open-ended answers, and output strictly True or False for verification. Answers were scored 0–5 on correctness, completeness, and conciseness by multiple LLM judges, then combined into F1-like and informativeness scores. To validate this automated scoring, they sampled 300 test questions, generated answers from every model, controlled for length, and had humans judge masked, randomized pairwise comparisons against the model judges.

Why This Matters

Impact on research. Benchmarks shape what the field optimizes for. By showing that text-only prompting beats page-image prompting for the same model and that surface metrics like ROUGE and BERTScore are actively misleading, this work argues that research-paper comprehension needs richer, multi-dimensional evaluation. The public dataset and taxonomy also give researchers a reproducible testbed for studying long-document reasoning, evidence grounding, and multimodal scientific understanding.

Real-world applications:

• Research copilots and literature assistants — products that summarize, compare, or answer questions about papers need exactly the skills this benchmark measures: method comparison, motivation analysis, and experimental reasoning.
• Automated peer-review triage — tools that flag missing baselines or unclear experimental setups must first be able to verify claims against a paper's content; the claim-verification results show how far current models are from that.
• Scientific knowledge extraction pipelines — building structured databases of methods, datasets, and results from published literature requires the fine-grained extraction that document-centric and RAG models currently perform poorly at.
• Model selection and procurement — organizations choosing between frontier LLMs, VLMs, or RAG stacks now have comparative evidence about which approach handles which type of scholarly question.

Industry relevance. The results matter directly for anyone building document AI: the consistent finding that image-based input degrades performance, that small specialized document models underperform despite domain fine-tuning, and that RAG failures often stem from retrieval errors rather than generation, all point to specific engineering priorities. Long context alone does not solve paper understanding.

Future Directions

• Extending beyond computer science. The benchmark currently covers CS and its subfields. The authors explicitly note that the pipeline could be adapted to life sciences, social sciences, and other domains where OpenReview-style public review data is available.

• Cross-document and multi-paper reasoning. RPC-Bench evaluates single-article comprehension by design. Moving to synthesis across papers — literature reviews, contradiction detection, cross-paper method comparison — is the natural next step and a much harder task.

• Closing the multimodal gap. The 4.74–36.1% drop when switching from text to page images is a concrete, measurable target. Better methods for grounding answers in figures, tables, and formulas — and for integrating them with long textual context — are clearly needed.

• Improving correctness and completeness, not just conciseness. Fine-tuning experiments show models readily learn to be brief but struggle to become accurate and complete. Understanding how to teach genuine scholarly reasoning, rather than surface-level answer shaping, remains an open problem.

Target Audience

This paper is most valuable to NLP and machine learning researchers working on document understanding, long-context reasoning, multimodal models, and evaluation methodology. It is also useful for practitioners building research tools, literature search systems, or scholarly document processing pipelines who need to choose between LLM, VLM, and RAG architectures and want evidence about their relative strengths. Benchmark designers and anyone concerned with LLM-as-a-Judge reliability will find the judge-alignment ablations directly applicable beyond this specific dataset.

Authors’ abstract

Understanding research papers remains challenging for foundation models due to specialized scientific discourse and complex figures and tables, yet existing benchmarks offer limited fine-grained evaluation at scale. To address this gap, we introduce RPC-Bench, a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers, containing 15K human-verified QA pairs. We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts. We also define an elaborate LLM-human interaction annotation framework to support large-scale labeling and quality control. Following the LLM-as-a-Judge paradigm, we develop a scalable framework that evaluates models on correctness-completeness and conciseness, with high agreement to human judgment. Experiments reveal that even the strongest models (GPT-5) achieve only 68.2% correctness-completeness, dropping to 37.46% after conciseness adjustment, highlighting substantial gaps in precise academic paper understanding. Our code and data are available at https://rpc-bench.github.io/.