PaperScorer: How Predictable Is Paper Acceptance?

AI summary

Overview

Research area: Machine learning for peer-review modeling and automated research evaluation (paper acceptance prediction, multimodal document understanding).

Technical level: Intermediate. The methods are standard supervised fine-tuning and reinforcement learning of vision-language models, but the paper assumes familiarity with conference review processes and evaluation metrics such as AUC and calibration error.

Scope: A controlled empirical study of how much of a paper's final accept/reject decision can be predicted from the anonymized paper alone, using two balanced corpora (ICLR and arXiv-derived) and a family of 3B–14B Qwen-based models called PaperScorer.

What This Paper Is About

Peer review decides whether research is accepted, but it is noisy: in the NeurIPS '14 and '21 consistency experiments, roughly half of papers accepted by one committee would not have been accepted by the other. The authors ask a narrow, measurable question: given only the paper, stripped of author identity and formatting shortcuts, how predictable is the final decision? They build PaperScorer, a family of models trained only on the final accept/reject label, and compare a text view (markdown) against a visual view (page screenshots) of the same papers.

Key Contributions

1. Two fully anonymized, balanced corpora. An ICLR corpus of 25K submissions from OpenReview (evaluated on '25–'26) and an arXiv corpus of ML conference papers reconstructed from LaTeX source, with an 83K training set (arXiv) and a 25K set proportional to ICLR (arXiv-m). Both are balanced 50/50 accept/reject, split 85/5/10 across train/validation/test.

2. PaperScorer models across text and vision. Supervised fine-tuning of Qwen2.5 text models (3B, 7B, 14B) and Qwen2.5-VL vision models (3B, 7B), plus reasoning-style reinforcement-learning variants trained with GRPO, all predicting a single decision-only target.

3. An estimated Bayesian ceiling. A Beta-Bernoulli fit loosely derived from the NeurIPS consistency experiment gives latent acceptance probability with (α, β) ≈ (0.39, 1.10), implying Bayes-optimal accuracy of approximately 83.3% under the original acceptance distribution and 80.2% on balanced evaluation, used only as a soft reference point.

4. A diagnostic analysis of where the signal lives. Breakdowns by reviewer rating, confidence calibration, model scale, data scale, and input compression, showing that predictability is concentrated at the rating extremes.

Main Findings

• Vision beats text and frontier models. PaperScorer-Vision reaches 68.3 accuracy on ICLR, 2.7% ahead of the best frontier vision model (Gemini-3-Flash at 65.6), and 77.5 on arXiv, 9.0% ahead of the best frontier baseline. Gains in accuracy and AUC are significant under a one-sided paired bootstrap (p < .05).

• Simple features already carry signal. A logistic-regression baseline on hand-crafted paper features reaches 61.1% on ICLR and 62.5% on arXiv. Accept-associated features are mostly length, references, and figures; reject-associated features are more lexical (for example sentence length and rare-word ratio).

• Middle-rated papers stay unpredictable. PaperScorer is strongest on the lowest and highest rated papers. Frontier models reliably flag only the lowest-rated papers, while PaperScorer is accurate on both low and high. Across all evaluated models, accuracy remains near chance for papers with intermediate reviewer ratings.

• Confidence is calibrated. After temperature scaling on validation data, confidence matches accuracy on the test sets, so the model's most confident predictions are also its most accurate. Calibration is reported only for PaperScorer and Jev, since frontier ratings are 1–10 review-style scores rather than probabilities.

• Direct binary supervision beats reasoning RL. ICLR-trained PaperScorer-Vision (RL) reaches 55.0 accuracy and PaperScorer-Text (RL) 56.1, trailing prompting baselines; the paper attributes this to learning from a single noisy accept/reject bit.

• Model size barely matters, data size does. Relative to matched 7B checkpoints, 3B and 14B variants move accuracy by small amounts with no consistent direction, and nearly every change is statistically insignificant at a 95% paired bootstrap CI. The largest arXiv gain comes from the 3.4× data multiple between arXiv-m and arXiv.

• The signal survives severe compression. Tiling ten pages into one 2380×1512 image cuts vision tokens by about 63% (roughly 12,420 to 4,590) and drops ICLR accuracy only 2.2% (68.3 to 66.1, AUC .736 to .704). OCR retention falls from about 80% on per-page screenshots to about 30% on the compressed panel, while layout is preserved.

• Coding agents rank but decide poorly. GPT-5.5 (Codex Agent) and Sonnet-4.6 (Claude Agent) have among the strongest baseline AUCs (for example .711 and .739 on ICLR and arXiv for Claude) but lower accuracy than the best non-agent baselines (55.3 and 61.2 on ICLR).

• Performance drifts over time. Accuracy and AUC drop sharply from '25 to '26 across PaperScorer, frontier models, and DeepReview, suggesting a shared shift in the evaluation distribution.

• No clear evidence of memorization in GPT-5.5. GPT-5.5 recognizes many test papers, but the decisions it claims to recall are at chance once its accept bias is accounted for, it cannot regenerate abstracts from titles (ROUGE-L at noise level), and its accuracy on recognized papers matches its accuracy on unrecognized ones.

Methodology in Plain English

The authors treat prediction as a text-generation task: the model is shown a paper and must output either "Accept" or "Reject." Because "Accept" and "Reject" are single tokens in the Qwen2.5 vocabulary, the normalized probability over those two tokens gives a continuous acceptance score in addition to a hard decision.

Building clean data is the bulk of the work. For ICLR, they pull latest-rebuttal PDFs from OpenReview so that accepted and rejected papers are obtained the same way, exclude ICLR '24 because its accepts are only available as camera-ready versions, and filter to full main-conference submissions. For arXiv, they start from 434,579 ML papers, pull 434,443 LaTeX source tarballs, compile each with its venue's anonymization style file, and define an "implied reject" as a paper that uses a covered venue template, passes the main-conference filters, compiles anonymously, and does not appear in the proceedings — a noisier but more diverse source. Papers must be anonymous, main-conference, and processed identically across labels; a references section and between six and 30 main content pages are required. All papers run through MinerU normalization, with headers, line numbers, author-identifying spans, and everything from the references onward removed or whitened.

Text models see markdown up to 24,480 tokens; vision models see page screenshots at up to 1,003,520 pixels (1280 × 28²), roughly 1,240 vision tokens per page, averaging about 12,420 tokens per paper. Training uses balanced classes because natural-rate training makes models predict rejection far too often under the native threshold, and validation-tuned thresholds overcorrect in the other direction.

Baselines include mini and large frontier reasoning models (Gemini-3-Flash, GPT-5.4-Mini, Gemini-3.1-Pro, GPT-5.5) on both views, coding-agent harnesses, a decision model called Jev, and review-trained DeepReview-7B/14B. Because many baselines favor one class, the authors also tune a threshold on each baseline's score using a small held-out sample and use the thresholded score when it helps.

Why This Matters

The paper measures a quantity that automated research actually needs: how much of a human judgment is contained in the artifact itself. As LLM agents begin proposing ideas and running experiments, they need a way to judge whether a result is good, and benchmarks cover only part of what peer review rewards. Knowing that paper content supports roughly 68–77% accuracy while a balanced Bayesian ceiling from historical consistency data sits near 80.2% tells us both that the signal is real and that a substantial remainder is noise from reviewer assignment, interpretation, and aggregation.

Real-world applications:

• Submission triage. A calibrated score could flag the clearest accepts and rejects for faster handling, while routing middle-rated papers to more careful human review.
• Agentic research loops. An acceptance-style scorer can act as an evaluator inside self-improving research agents that have no benchmark to optimize against.
• Literature search and ranking. The acceptance score ranks accepted above rejected papers and correlates with reviewer ratings on ICLR, so it can order papers without a fixed threshold.
• Review-process auditing. Comparing model confidence against outcomes can surface where decisions are least determined by content, which is where reviewer noise is largest.

Industry relevance: publishers and conference platforms, AI labs building self-improving research agents, and academic search and recommendation products all depend on cheap, calibrated judgments of research quality at scale. The finding that a 3B model performs comparably to a 7B model matters for deployment cost.

Future Directions

• The middle-rated papers. Accuracy stays near chance for intermediate reviewer ratings across every model tested. Whether this reflects irreducible decision noise or a signal the current representations miss is unresolved.

• Temporal drift and retraining. Accuracy drops from '25 to '26, and training only through '25 transfers poorly to '26. How often a deployed predictor must be recalibrated, and whether drift can be anticipated, is open.

• Better use of weak supervision. RL with a single noisy accept/reject bit underperformed plain supervised fine-tuning. Richer reward signals, or combining decisions with the review text the authors deliberately excluded, may recover some of the gap.

• Separating visual gestalt from readable text. The compressed-panel result preserves layout while cutting recoverable text from about 80% to about 30%, but the authors state they cannot fully isolate visual presentation from text that remains readable.

Target Audience

Researchers working on automated peer review, LLM-based research agents, and multimodal document understanding; conference organizers and publishers interested in decision-support tools; and machine learning practitioners who want a careful example of how to build a label-prediction dataset without shortcut leakage. Readers curious about the limits of predicting human judgment from artifacts alone will also find the calibration and rating-stratified analyses useful.

Authors’ abstract

Automated research is largely evaluated by benchmark performance, but the value of research rarely reduces to a single score. Humans judge a work's holistic contribution through peer review, which is hard to model because it is noisy and depends on many factors. However, what if, given only the paper, we could predict its peer-review decision directly? We introduce PaperScorer, a family of models that predict a paper's accept or reject decision from its extracted text or page screenshots, training only on the decision after removing author identity and formatting shortcuts. Vision models are the most predictive (68% accuracy on ICLR papers and 77% on ML papers from arXiv), beating text-only and frontier reasoning models. Unlike plain text, page images retain figures, layout, and presentation. This signal is robust, surviving smaller models and even compressing the whole paper into a single image. But accurate predictions are not evenly spread. PaperScorer is strongest on the lowest and highest rated papers, while papers rated in the middle stay unpredictable for our models and frontier systems alike. Additionally, its confidence is calibrated, so its most confident predictions are also its most accurate. Overall, PaperScorer shows how much of the decision a paper carries on its own, marking a first step toward AI-driven research evaluation.