Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery

AI summary

Overview

• Research area: Natural Language Processing / AI evaluation, applied to biomedical scientific discovery (LLM agents, benchmark construction, contamination-free evaluation).
• Technical level: Intermediate.
• Scope: The paper introduces DBench-Bio, a dynamic, fully automated benchmark that uses newly published biology abstracts to test whether large language models can derive genuinely new biological knowledge rather than recall what they were trained on.

What This Paper Is About

Existing benchmarks for measuring AI's ability to do science use static, fixed datasets, which means models have likely already seen the answers during training, and the benchmarks go stale as new models are released. The authors build a benchmark that continuously pulls fresh questions from papers published after a model's release date, so a model cannot score well by remembering. The goal is to measure "derivation-based knowledge discovery": the ability to recombine established knowledge to infer findings that are genuinely new to the model.

Key Contributions

1. A methodology for dynamic, fully automatic benchmark construction, designed specifically for knowledge discovery, with temporal separation as a core requirement (evaluation knowledge must post-date the model's release, and tool-based retrieval must be restricted to pre-collection dates).
2. DBench-Bio, an instantiated, monthly-updated benchmark covering 12 biomedical sub-domains, sourced from JCR "Biology & Biochemistry" Q1 journals, with two temporal snapshots (December 2025 and January 2026).
3. An automated three-stage pipeline (data acquisition, QA extraction, QA filter) validated against three human experts using the Alt-test framework, showing LLM-generated annotation can substitute for expert annotation.
4. Extensive empirical evaluation of SOTA models across three variants (base LLMs with and without thinking, LLM with tools, ReAct agent, and agent workflow), with quantitative results plus a taxonomy of five failure modes.

Main Findings

• Low overall performance: Aggregate scores across all evaluated models remain low on the 1–5 scale, which the authors read as evidence that current LLMs have not mastered deriving new knowledge. The benchmark instances were built for December 2025 (4,231 QA pairs total) and January 2026 (3,791 QA pairs total).
• Thinking helps some models, not others: Explicit thinking strategies gave significant gains for Kimi-K2, GPT-5, GPT-5-Mini and GPT-5.2, but negligible gains for GLM-4.6, GLM-4.7 and DeepSeek-V3.2.
• Tool use alone gains little: Adding a restricted PubMed retrieval tool did not yield significant improvements, which the authors attribute to retrieved information largely overlapping with models' internal knowledge.
• Agent architectures help: Both ReAct (iterative reasoning-acting loop) and a multi-agent workflow (Planner, Tool Caller, Reasoner, Reporter, Critic) improved performance, with only marginal differences between the two paradigms.
• Agents scale with backbone quality: In a randomly sampled subset of 1,000 instances, agent-based methods consistently outperformed their corresponding backbone models, and performance improved as the backbone model got stronger. GPT-5.2 outperformed the agent methods, which used GPT-5-Mini as backbone.
• Mathematical & Computational Biology is the hardest sub-domain: Performance dropped noticeably for all models in this category, pointing to weakness in complex mathematical reasoning and computational simulation.
• Basic knowledge is necessary but insufficient: Thinking models exceeded 90% accuracy on MMLU-Pro (Biology) with negligible variance, yet still performed poorly on DBench-Bio. Baichuan-M2 was the weakest on both.
• Rankings misalign across benchmarks: Gemini-3-Flash led on MMLU-Pro (Biology) but performed relatively worse on DBench-Bio, suggesting static knowledge retention does not translate into discovery ability.
• Five failure modes: Mechanism Error (plausible but wrong mechanism), Information Omission (partial mechanism, missing required detail), Generic Mechanism Substitution (textbook mechanisms instead of the specific experimental finding), Refusal to Answer, and Overconfident Reasoning (skipping the tool step and hallucinating from internal knowledge).

Methodology in Plain English

The pipeline runs in three stages. First, the authors collect abstracts only from journals in the JCR "Biology & Biochemistry" category ranked in the JIF Q1 quartile, using PubMed's "Entry Date" rather than "Publication Date" so a paper's public availability is approximated accurately. A domain expert picked 12 cell-biology-related sub-domains and excluded less relevant ones such as Agronomy and Zoology. Critically, only abstracts published after the target model's release date are crawled, preventing data contamination.

Second, an LLM (for example, DeepSeek-V3.2-thinking) turns each abstract into one question-answer pair, where the question is a scientific hypothesis inquiry (for example, "Does X regulate Y phenomenon?" or "What is the mechanism behind Z effect?") and the answer is the core conclusion, written as concise bullet points. The pairs deliberately avoid fine-grained parameters and numerical values, targeting causal inference and mechanistic explanation instead.

Third, an LLM judge scores each pair from 1 to 5 on relevance (alignment with the target sub-domain), clarity (linguistic precision, filtering out context-dependent phrases like "based on the text"), and centrality (whether the pair captures the abstract's primary finding rather than peripheral detail). Only pairs scoring Relevance ≥ 4, Clarity ≥ 5 and Centrality ≥ 5 are kept.

For quality assurance, three biology experts manually assessed 200 QA pairs following the Alt-test protocol. Winning rates for relevance, clarity, centrality and evaluation were 0.90, 1.00, 1.00 and 0.90; advantage probabilities were 0.94, 0.96, 1.00 and 0.82; Gwet's AC1 inter-annotator agreement was 0.98, 0.93, 0.99 and 0.78.

Evaluation uses an LLM-as-a-Judge setup with a 1–5 Likert scale, where 5 means the answer is correct and comprehensive and 1 means it is completely incorrect, hallucinated or irrelevant. DeepSeek-V3.2 was used both for QA filtering and answer evaluation. To keep evaluations balanced, any sub-domain yielding more than 500 QA pairs was randomly capped at 500.

Why This Matters

Evaluation integrity is the central issue: if a benchmark's answers sit in a model's training data, high scores may reflect memorization rather than discovery, which makes it impossible to tell whether AI is actually contributing new scientific insight. DBench-Bio's design makes the evaluation a moving target, so scores should stay meaningful as models are retrained and re-released.

Real-world applications:

• Drug discovery and target identification: testing whether AI can infer mechanisms, such as how a compound inhibits a cancer pathway, from the latest literature.
• Hypothesis generation for wet-lab research: screening candidate hypotheses generated by AI co-scientist systems before committing experimental resources.
• Benchmarking scientific AI agents: providing a contamination-resistant yardstick for evaluating agents such as AI co-scientist, Biomini and BioLab.
• Cross-disciplinary evaluation: the pipeline generalizes by swapping the JCR source category, enabling analogous benchmarks in other scientific fields.

Industry relevance: pharmaceutical and biotech R&D groups increasingly pilot LLM agents for literature review and hypothesis triage, and AI labs building scientific agents need trustworthy evaluations to demonstrate capability. The paper's core warning is that scaling pre-training data alone does not appear to produce discovery ability, so vendors and adopters may need specialized mechanisms rather than larger models.

Future Directions

• Broaden model coverage: the authors state that time and cost constraints led them to evaluate only a representative set of models instead of all available SOTA LLMs, and call for evaluating a broader range of models.
• Extend the pipeline to other domains: replacing the JCR source category would produce a family of benchmarks for cross-disciplinary evaluation of knowledge discovery.
• Close the memorization-to-discovery gap: the paper argues current models rely on memorization and pattern matching, and that knowledge discovery requires specialized mechanisms rather than more pre-training data — how to build such mechanisms is left open.
• Address the identified failure modes: mechanism errors, information omission, generic mechanism substitution, refusal, and overconfident reasoning are unmitigated, and it is not yet clear whether better agent scaffolding, different retrieval strategies, or new training objectives would resolve them.

Target Audience

Researchers and practitioners working on LLM evaluation, scientific AI agents and automated benchmark construction; biomedical and pharmaceutical data scientists assessing whether LLMs can contribute to hypothesis generation; and AI lab teams who need contamination-free benchmarks for model development decisions. Readers should be comfortable with benchmark design terminology and LLM evaluation protocols.

Authors’ abstract

Recent advancements in Large Language Model (LLM) agents have demonstrated remarkable potential in automatic knowledge discovery. However, rigorously evaluating an AI's capacity for knowledge discovery remains a critical challenge. Existing benchmarks predominantly rely on static datasets, leading to inevitable data contamination where models have likely seen the evaluation knowledge during training. Furthermore, the rapid release cycles of modern LLMs render static benchmarks quickly outdated, failing to assess the ability to discover truly new knowledge. To address these limitations, we propose DBench-Bio, a dynamic and fully automated benchmark designed to evaluate AI's biological knowledge discovery ability. DBench-Bio employs a three-stage pipeline: (1) data acquisition of rigorous, authoritative paper abstracts; (2) QA extraction utilizing LLMs to synthesize scientific hypothesis questions and corresponding discovery answers; and (3) QA filter to ensure quality based on relevance, clarity, and centrality. We instantiate this pipeline to construct a monthly-updated benchmark covering 12 biomedical sub-domains. Extensive evaluations of SOTA models reveal current limitations in discovering new knowledge. Our work provides the first dynamic, automatic framework for assessing the new knowledge discovery capabilities of AI systems, establishing a living, evolving resource for AI research community to catalyze the development of knowledge discovery.