Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies

AI summary

Overview

Research area: AI for science; evaluation of large language models on scientific idea generation and literature-grounded reasoning.

Technical level: Intermediate. The core question is intuitive, but the paper contains a formal benchmark definition, multi-agent selection procedure, judge-recusal rules, bootstrap statistics, and contamination stratification.

One-sentence scope: The paper introduces a blind, time-cut benchmark that hides a published paper's own idea and all contemporaneous literature, gives models only that paper's pre-publication bibliography, and measures how often they can reconstruct what the paper actually proposed.

What This Paper Is About

Most evaluations of LLM-generated research ideas ask whether an idea is novel, interesting, or preferred by humans. This paper asks a harder, more diagnostic question: given only the references a paper cited before it was published, can a model recover what that paper actually proposed? To test this, the authors build a "Reconstruction" benchmark in which the seed paper, its title, and its abstract are withheld from the model, and an independent LLM judge later compares the model's hypotheses against the held-out ground-truth idea.

Key Contributions

1. The Reconstruction benchmark. A time-cut, anti-leakage protocol for measuring idea recovery from pre-publication bibliographies across machine learning and five Nature-family domains (Astronomy, Chemistry, Materials, Medicine, Physics). The design uses a hard temporal citation cutoff, anonymous reference IDs, information isolation from the seed, and frozen per-paper bibliographies shared across conditions.

2. Single-model baselines. Seven frontier models evaluated on 643 papers. The best six-domain average is 13.3% ± 2.3% (Claude-Opus-4.8), with domain scores typically in the 3.4–15.0% band.

3. A reference-only multi-agent pipeline. Cross-model review plus Swiss-tournament selection over aligned hypothesis slots raises Match rates to 22.9–41.6% across all six domains, with mean 36.0% and an observed 2.4× lift over the best dagger single-model baseline (top 4 proposers scored only by the other top 3 models). No external web search is used; the authors credit the full selection pipeline rather than collaboration alone.

4. Contamination strata. The paper stratifies results by earliest public date versus each model's knowledge cutoff, treating "date ≤ knowledge cutoff" as an upper bound on corpus reachability rather than a demonstration of instance-level memorization. The qualitative pattern holds on the n = 236 papers first public after 2026-03-22, with Match rates 19.7–41.6%, mean 31.1%, and lift still 2.4× versus the same dagger baseline.

Main Findings

• Single models recover ideas only rarely. Across 643 evaluated papers and six domains, seven frontier models achieve Match rates spanning 3.4–15.0%. The best six-domain average is 13.3% ± 2.3% (Claude-Opus-4.8).

• Model ranking and per-domain top scores. Claude-Opus-4.8 leads the six-domain average (13.3% ± 2.3%); GPT-5.6-Sol-Pro follows at 12.8% ± 2.4%. GPT-5.6-Sol-Pro records the best Chemistry cell (15.0% ± 3.8) and Medicine cell (14.6% ± 2.7) among the seven default models; Claude-Opus-4.8 records the best ML (8.2% ± 2.4), Astronomy (14.7% ± 3.1), Materials (14.0% ± 3.7), and Physics (15.0% ± 3.2) cells. The weakest overall average is Qwen3.7-Max at 5.9% ± 1.3%.

• The multi-agent pipeline produces a large jump. Cross-model review plus Swiss selection reaches 22.9% (ML), 36.5% (Astronomy), 38.4% (Chemistry), 40.1% (Materials), 41.6% (Medicine), and 36.4% (Physics), averaging 36.0% ± 6.1% across domains.

• Lift ratios are narrow and consistent. Against the best dagger (top 4, other-top-3 judges) single-model scores, observed ratios are 2.5× (ML), 2.4× (Astronomy), 2.3× (Chemistry), 2.6× (Materials), 2.6× (Medicine), and 2.3× (Physics); the mean of these ratios is 2.4×.

• Paper-level bootstrap supports the lift. On the pooled 643 papers, overall lift is 2.4× with a 95% percentile CI of [2.3, 2.6], using B = 2000 replications. A paper-wise paired comparison against the per-paper max among the four dagger top-4 models favors multi-agent on 343 versus 160 papers, with 140 ties (two-sided sign test p < 10⁻¹⁵).

• success@5 improves much less than per-hypothesis Match rate. Multi-agent success@5 is 57.1% overall versus 55.1% for the domain-best dagger single. In Chemistry (59.0% vs. 60.6%) and Medicine (65.0% vs. 66.2%), multi-agent success@5 is slightly below the domain-best dagger. The authors conclude the Match lift comes mainly from recovering more matching hypotheses on papers that already have at least one hit, not from covering many additional papers.

• The pipeline beats Default-grid slot selection but not an unconstrained oracle. On the frozen Default top-4 4 × 5 grid scored by dagger peers, the best fixed single model reaches 14.6% Overall, a seeded slot-wise random selection (A) reaches 12.3%, the pool mean over all 20 hypotheses (B) reaches 12.6%, a slot oracle with match-label peeking (C) reaches 30.8%, an unconstrained oracle taking the five highest of all 20 (D) reaches 44.5%, and multi-agent reaches 35.6%. Because MA > C in every domain, the result is not an oracle violation; the overall ordering is C < MA < D.

• Hypotheses are shorter than seed texts, and multi-agent is closer to seed length than Default. Overall, Default hypotheses average 56 words (median 49), multi-agent 114 words (median 95), and seed papers (title + abstract) 191 words (median 183). The authors flag length/presentation as a possible confounder pending controls that hold hypothesis length fixed.

• The pattern survives the latest-cutoff slice. Splitting at 2026-03-22 (Kimi's inferred knowledge cutoff) gives n≤ = 407 versus n> = 236. On the later slice, multi-agent Match rates are 19.7% (ML), 33.0% (Astronomy), 31.7% (Chemistry), 31.8% (Materials), 41.6% (Medicine), and 28.7% (Physics), averaging 31.1% ± 6.5%, versus a best dagger single-model average of 12.4% on the same slice. Domain lift ratios range from 1.7× to 3.5×, averaging 2.4×.

• Contamination is bounded, not resolved. Most seeds are from 2026 (498/643 = 77.4%). The share of papers with earliest public date ≤ knowledge cutoff ranges from 5.1% (Gemini, KC 2025-01-31) to 63.3% (Kimi, inferred KC 2026-03-22). The Δ = Match(≤) − Match(>) gap is positive for all seven Default models and for multi-agent (Default mean +2.7%; multi-agent +8.5%), concentrated in Chemistry, Materials, Physics, and Astronomy, with ML often flat or reversed. The authors emphasize these compare two disjoint paper sets of unequal size and different domain/year mix, and should not be read as measuring how much memorization raises Match rate.

• The non-matched design is acknowledged. Multi-agent selects 5 hypotheses from 20 candidates (4 models × 5 slots) while Default reports 5 of 5 from one model, so part of the gain may reflect inference-time scaling, and the two settings are scored by different judge panels. The authors report the comparison as an observed association for the full pipeline.

Methodology in Plain English

For each "seed" paper, the researchers reconstruct the reading list that existed before the paper appeared. They take the paper's bibliography, keep only references published strictly before the seed's publication date, strip identifying information, and re-label the entries as opaque IDs such as ref-001, each with its title and abstract. The seed paper itself and any same-day or later literature are removed entirely.

Models then get one job: propose five distinct hypotheses with supporting reference IDs, using only that frozen blind bibliography, with no web search. Each hypothesis is later graded by a separate LLM judge that sees the seed's real title and abstract alongside the hypothesis title and summary, and returns a binary match or no-match. The headline metric, Match rate, is the fraction of a paper's five hypotheses labeled as matching.

The multi-agent condition takes the four strongest single models by six-domain average — Claude-Opus-4.8, GPT-5.6-Sol-Pro, Kimi-K3, and GLM-5.2. Each produces five hypotheses in one call. The authors align hypotheses by slot, so slot 1 across the four models is one candidate group, producing 20 candidates total. Other models review each candidate (the proposer recuses itself) and the proposer may revise in response, using only the blind references. Each slot then runs a Swiss tournament with conflict-of-interest recusal and presentation-order debiasing. The five slot champions form one multi-agent case and are scored by the final judge independently of the earlier review and tournament calls.

To stop the seed from leaking into the prompt, the design uses a hard temporal cutoff, information isolation, anonymous reference IDs, frozen per-paper bibliographies reused across conditions, required evidence binding to anonymous references, and a rule that the judge model must differ from the proposer. Dataset construction drew 879 titles from six sources (the full ICML 2026 Oral program at N = 168, plus Nature Astronomy, Nature Chemistry, Nature Materials, Nature Medicine, and Nature Physics pages scraped on 14 July 2026), resolved them through Semantic Scholar with fallbacks to OpenReview, Crossref, OpenAlex, and arXiv, and required each retained seed to have a usable publication date and at least three references published strictly before T₀. Of 879 titles, 53 never entered a Default run, leaving 826 launched and 745 completed; 21 more were dropped for having fewer than five hypotheses from at least one top-4 model, yielding the reported set of 643 papers.

Why This Matters

Reconstruction turns an abstract question — do models actually understand a literature well enough to anticipate its next move — into a measurable, blind, judge-scored task. Because the benchmark cuts off all contemporaneous and future information and anonymizes references, it separates genuine bibliographic reasoning from the easier path of recognizing a familiar paper. That distinction matters for anyone building systems that claim to reason over scientific literature rather than recall it.

The headline result is sobering for single-model use: even the strongest frontier models match the true idea for only a small fraction of hypotheses. Equally notable is that a reference-only ensemble pipeline nearly doubles to more than doubles those rates without any web search, while still falling well below an infeasible oracle that peeks at the answer labels. The paper is also unusually explicit about what its numbers do and do not show — it labels the multi-agent gain an observed association, presents zero-compute bounds to isolate candidate-count effects, and stratifies results by knowledge cutoff rather than claiming the benchmark is contamination-free.

Potential real-world applications (framed as implications of the work, not claims evaluated in the paper):

• Literature-grounded research assistants. Tools that help scientists generate and prioritize hypotheses from the papers they have already cited, rather than from a model's memorized priors.
• Gap and opportunity analysis. Identifying which directions a body of prior work makes plausible, using the same pre-publication bibliography as evidence.
• Model and pipeline evaluation. Using Reconstruction-style blind match scoring as an internal yardstick when choosing between single-model prompting and multi-agent review-and-select pipelines.
• Multi-agent system design. Informing how review, selection, and slot alignment should be structured when ensembles are used for reasoning tasks, given the paper's finding that selection from many candidates drives much of the gain.

Industry relevance: the evaluated models come from Anthropic, OpenAI, MoonshotAI, Z.ai, Google, DeepSeek, and Qwen, and the work speaks directly to how inference-time scaling, ensemble selection, and LLM-as-judge evaluation are deployed in practice. It also surfaces concrete operational issues, such as provider content-filter refusals that silently drop proposer outputs (Claude-Opus-4.8 returned empty outputs with finish_reason=content_filter on 4 audited Medicine papers, and Materials lost one paper to a provider refusal), which matter for anyone running large-scale automated evaluation.

Future Directions

• A matched best-of-20 control. The authors state that the Default-versus-multi-agent comparison is not candidate-matched and that the zero-compute bounds do not replace a matched best-of-20 control with shared review, Swiss selection, and identical judge panels.
• Candidate-matched controls in general. The paper explicitly leaves candidate-matched controls to follow-up work, along with ablations needed to separate the selection pipeline's contribution from raw candidate count.
• Length-controlled evaluation. Hypothesis length is an uncontrolled variable (Default 56 words on average vs. multi-agent 114 vs. seeds 191), and the authors call for controls that hold hypothesis length fixed.
• Instance-level contamination testing. Date ≤ knowledge cutoff is only an upper bound on corpus reachability; the paper notes that nonzero Match on the > slice is compatible with bibliography-conditioned inference and with remaining confounds, leaving direct memorization tests open.
• Judge-panel mismatch. Because Default and multi-agent are scored by different judge panels, the authors note this caveat remains unresolved by the current bounds.

Target Audience

Researchers and engineers working on LLM-based scientific ideation, automated discovery pipelines, and multi-agent inference-time scaling; benchmark designers interested in temporal cutoffs, anti-leakage protocols, and LLM-as-judge evaluation; and practitioners selecting between single-model and ensemble approaches for literature-grounded hypothesis generation. Readers need only a general familiarity with language model prompting and evaluation to follow the core argument, though the statistical appendices assume comfort with bootstrap confidence intervals and paired tests.

Authors’ abstract

Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature, and asks models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea. A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea. Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx. 3-15%). We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search. Cross-model review plus tournament selection raises Match rates to approx. 23-42% across all six domains, which is an observed approx. 2.4x lift over the best single-model baseline. This draft reports the protocol, anti-leakage design, and current results as an arXiv timestamp.