HeurekaBench: A Benchmarking Framework for AI Co-scientist

AI summary

Overview

Research area: Machine learning evaluation, LLM-based AI agents, and computational biology (single-cell genomics). Technical level: Intermediate — the paper is accessible to readers familiar with LLMs and benchmarks, and includes domain-specific details for single-cell biology. Scope: This paper introduces HeurekaBench, a framework that builds open-ended benchmark questions directly from published scientific papers and their code, then uses it to evaluate and analyze AI agents acting as co-scientists in single-cell biology.

What This Paper Is About

Existing benchmarks for scientific AI agents test narrow skills like recalling facts or answering a single computational question, but real scientific discovery requires autonomously exploring a dataset, planning multi-step analyses, and drawing new interpretations. The authors build HeurekaBench, a framework that turns published research studies and their code repositories into open-ended research questions with validated ground-truth answers, so AI agents can be tested on the kind of exploratory work a genuine co-scientist would perform.

Key Contributions

1. A benchmark construction framework (HeurekaBench) that grounds question generation in published studies and their code repositories, using a multi-LLM pipeline with human verification to filter out unverifiable or hallucinated insights.
2. Two complementary question formats built from the same validated insights: open-ended research questions (OEQs) for realistic evaluation and multiple-choice questions (MCQs) for rapid development-time checks.
3. sc-HeurekaBench, a concrete single-cell biology benchmark containing 50 OEQs and 50 MCQs derived from 41 validated insights across 13 papers (plus a smaller sc-HeurekaBench-TU variant targeting domain-specific tool use).
4. A new LLM-as-a-judge evaluation scheme that decomposes both agent responses and ground-truth answers into atomic facts, rewarding dataset-backed reasoning over memorized knowledge.

Main Findings

• Insight extraction works well: The pipeline's InsightExtractor module generated insights strongly related to expert-curated findings in 44 of 50 FlyBase cases and 14 of 21 BixBench cases, showing it can identify expert-level findings from raw papers.
• Code matching is effective but imperfect: The CodeDescriber and CodeMatcher modules correctly retrieved 74.6 percent of relevant code files on average, enough to support downstream workflow generation.
• Biomni leads among existing agents: On the reduced sc-HeurekaBench-Lite subset, Biomni scored highest on both open-ended questions (2.31/5) and MCQs (50 percent accuracy), outperforming BixBench-Agent (2.34/5, 44.4 percent) and CellVoyager (2.03/5, 27.8 percent) on the flexible-loop side.
• Planner choice matters a lot: Claude-4-Sonnet as planner scored 2.58/5 on open-ended questions, well above the best open-source planner GPT-OSS-120B at 2.08/5. Larger and "thinking"-style open models consistently improved over their smaller or non-reasoning counterparts.
• Critic placement is critical: Adding an end-of-loop critic raised GPT-OSS-120B's average score from 2.04 to 2.49, closing much of the gap to Claude-4-Sonnet, with the biggest gains on low-scoring questions (plus 0.6 points). A critic placed right after planning instead degraded performance.
• Retriever module is essential for tool-based tasks: On sc-HeurekaBench-TU, disabling the retriever dropped GPT-OSS-120B's score from 2.15 to 1.56, showing agents struggle to pick the right domain tools without it.
• Judge scores align with human experts: GPT-4o judge scores differed by at most 1 point from human expert ratings on 92-96 percent of questions, with Spearman correlation around 0.90-0.93.

Methodology in Plain English

The authors start by selecting recent papers from Nature and Cell that publish both single-cell RNA-seq data and open-source analysis code. They run a pipeline of LLMs to do four jobs in sequence: extract high-level insights from the paper text, convert code scripts into natural-language descriptions, match each insight to the code that supports it, and assemble those pieces into runnable multi-step workflows. Human reviewers then execute the workflows, make minor fixes, and keep only the insights whose computed output matches the numbers and figures reported in the paper. Each validated insight is reformulated into one open-ended question and one multiple-choice question with careful filtering to remove easy or hallucinated items. Agents then receive the dataset and question and are scored by an LLM judge that breaks answers into atomic facts and compares them to the ground truth. The authors also run ablation studies swapping out the planner model, adding a critic module at different points in the agent loop, and removing the retriever.

Why This Matters

This work changes how AI co-scientists are evaluated: instead of asking whether a model can follow a task instruction, it asks whether an agent can independently explore real data and produce findings that match published scientific results. It also shows that simple architectural changes — like adding a critic — can make open-source models competitive with closed-source frontier models, which has cost and reproducibility implications for research labs.

Real-world applications:

• Accelerating hypothesis generation in single-cell genomics labs, where researchers could use evaluated agents to screen datasets for patterns worth following up.
• Benchmarking commercial and open-source AI agents before deploying them in pharmaceutical or biotech discovery pipelines.
• Guiding agent architecture design by showing which components (planner, critic, retriever) actually move performance.
• Providing a template for building similar grounded benchmarks in other scientific domains such as chemistry, materials science, or astronomy.

Industry relevance: The findings about critic modules and retriever modules are directly actionable for companies building agentic AI products in science and engineering, and the benchmark framework offers a reliable way to compare vendors' models on realistic tasks rather than toy problems.

Future Directions

• Extend HeurekaBench to other scientific domains, which will require domain-expert validation at each stage.
• Use LLMs to automatically validate intermediate workflow steps against published studies, rather than relying solely on final-answer evaluation (the paper flags this as a current limitation).
• Keep sc-HeurekaBench continuously updated with new publications so it can evaluate newly trained models without contamination from memorized knowledge.
• Investigate how credit assignment for partial or intermediate workflow correctness could make evaluation more informative, especially for long multi-step analyses.

Target Audience

Researchers and engineers working on LLM-based agents, AI for science, or benchmark design will find the framework and evaluation scheme most useful. Computational biologists and single-cell analysts will benefit from the sc-HeurekaBench benchmark itself. Product teams at AI labs building scientific assistants should read the ablation results on planner, critic, and retriever design choices.

Authors’ abstract

LLM-based reasoning models have enabled the development of agentic systems that act as co-scientists, assisting in multi-step scientific analysis. However, evaluating these systems is challenging, as it requires realistic, end-to-end research scenarios that integrate data analysis, interpretation, and the generation of new insights from the experimental data. To address this limitation, we introduce HeurekaBench, a framework to create benchmarks with exploratory, open-ended research questions for experimental datasets. Each such question is grounded in a scientific study and its corresponding code repository, and is created using a semi-automated pipeline that leverages multiple LLMs to extract insights and generate candidate workflows, which are then verified against reported findings. We instantiate the framework in single-cell biology to obtain sc-HeurekaBench benchmark and use it to compare state-of-the-art single-cell agents. We further showcase the benefits of our benchmark for quantitatively analyzing current design choices in agentic systems. We find that the addition of a critic module can improve ill-formed responses for open-source LLM-based agents by up to 22% and close the gap with their closed-source counterparts. Overall, HeurekaBench sets a path toward rigorous, end-to-end evaluation of scientific agents, grounding benchmark construction in real scientific workflows.