AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents

AI summary

Overview

Research area: Automated benchmark construction and evaluation for AI agents that perform scientific data analysis (computational biology, materials science, clinical imaging).

Technical level: Advanced. The paper assumes familiarity with LLM agents, tool-use trajectories, benchmark saturation, and exact-match evaluation; the framing is conceptual but the experimental apparatus is specialized.

Scope: The paper introduces AutoSciBench, a framework that autonomously generates, judges, and iteratively refines scientific-agent benchmark tasks across three scientific domains, and reports that the resulting benchmarks are harder than human-curated seeds while receiving higher average quality ratings.

What This Paper Is About

Scientific benchmarks are expensive to build because they need domain expertise, and they go stale quickly as agent capabilities improve. AutoSciBench tests whether such benchmarks can instead be generated automatically and then continuously adapted based on how solver agents actually behave on the generated tasks. The goal is an evaluation loop in which solver failures and shortcuts feed back into task design, and lessons from past refinements inform the construction of new, harder tasks.

Key Contributions

1. A two-level task representation. Each evaluation target is split into a concept (a high-level plan given by the tuple c = (d, m, O): scientific domain, data modality, and required operations) and a recipe (a low-level plan specifying how the question, environment, and ground-truth answer are constructed and verified). This separation lets several different tasks target the same analytical capability, and lets the framework revise either the concrete task or the evaluation target itself.

2. Feedback-guided iterative refinement. Solvers attempt each implemented task, producing trajectories and answers; an LLM judge rates verifiability (whether the question and data support a unique, gradable answer) and difficulty (measured by empirical solve rate across trials). If a task is not both verifiable and hard, the framework revises the recipe or, after four consecutive recipe revisions without meeting both criteria, the underlying concept.

3. Experience-guided concept generation. Refinement histories are distilled into experience cards that record which designs permitted direct answers, which revisions changed solver behavior, and which issues remained unresolved. These cards accumulate into an experience memory that conditions the generation of subsequent concepts, alongside source priors drawn from existing benchmarks and scientific literature.

4. Empirical validation across three scientific fields. Using CompBioBench, MatTools, and MedCTA as seed benchmarks for computational biology, materials science, and clinical imaging, the authors generate 50 tasks per domain and show reduced solver accuracy, higher judged quality, and measurable contributions from both iterative refinement and experience reuse.

Main Findings

• Generated benchmarks are substantially harder than human-curated ones. Average solver accuracy drops by 22.4 percentage points in computational biology (from 79.8 on the human-curated benchmark to a 57.4 average on the Claude-Opus-4.8-generated benchmark) and by 25.5 percentage points in materials science (from 95.7 to 70.2). The abstract states the same 22.4 and 25.5 point reductions. GPT-5.6-Sol-generated benchmarks yield averages of 74.1 and 77.6 in those two domains.

• Materials-science tasks are near ceiling on the original benchmark but not on generated ones. Solver accuracy is concentrated near the ceiling on the human-curated materials-science benchmark (95.7 average) and varies substantially on the generated benchmarks, revealing performance differences that are less apparent on the original benchmark. The paper reports that human-curated clinical imaging accuracy is omitted from Table 1 because that benchmark uses LLM evaluation rather than exact match.

• Difficulty transfers beyond the generator-solver pair. Models other than the generator used during refinement also score lower on generated benchmarks than on the human-curated benchmarks.

• Stronger solving does not imply stronger task generation. Claude-Opus-4.8-generated benchmarks show lower average solver accuracy than GPT-5.6-Sol-generated benchmarks, even though GPT-5.6 Sol generally outperforms Claude Opus 4.8 as a solver across benchmarks. Average columns in Table 1 are 66.2 (Opus 4.8-generated) versus 75.4 (GPT-5.6 Sol-generated).

• Fixed generated benchmarks saturate with newer agents. Models released after the generators achieve substantially higher accuracy on the generated benchmarks: Opus 5.0 outperforms Opus 4.8 by 19.5 percentage points, and GPT-6 Astra outperforms GPT-5.6 Sol by 15.1 percentage points on their respective generated benchmarks.

• Generated tasks score higher on judged quality. Across all three domains and both judges (Claude Opus 4.8 and GPT-5.6 Sol), generated benchmarks receive higher average quality scores than human-curated ones on a 1-5 scale. Average scores: computational biology 3.18 (human), 3.93 (Opus 4.8), 4.28 (GPT-5.6 Sol); materials science 2.54, 3.87, 4.06; clinical imaging 2.23, 3.07, 4.02. GPT-5.6-Sol-generated tasks achieve the highest averages, with high fitness scores, while Opus-4.8-generated tasks score higher on usefulness in computational biology and materials science.

• Iterative refinement shifts the difficulty distribution. Initial tasks are predominantly easy, indicating the generating agent can solve most initial implementations; over subsequent refinement rounds the proportion of hard tasks increases.

• Refinement shifts tasks from direct output reading to interpreting intermediate evidence. In a UMI (unique molecular identifier) example, counting distinct UMI sequences recovers the correct original UMI count in the initial task, whereas the revised task injects UMI sequence errors that mimic real sequencing errors. The revised task requires separating error-induced sequence variation from true differences between original UMIs, using read support, as in UMI error-correction methods. Across all three solver trials on the revised task, agents treated distinct UMI sequences as separate original UMIs and answered incorrectly.

• Experience reuse makes subsequent concepts harder. In a materials-science ablation with Claude Opus 4.8, comparing equal budget (10 initial concepts, 50 tasks total, refinement retained), experience-guided generation reduces solver success from 66.7% to 35.0% and raises the proportion of hard tasks from 15.0% to 50.0%. A later analysis shows experience-guided concepts introduce a new domain, modality, and measurement while preserving higher-level solving behaviors; the resulting task was rated hard, whereas the task from the earlier concept was rated moderate.

• Structured plans matter more than editing tasks directly. In a materials-science comparison using Claude Opus 4.8, solver success decreases more substantially across refinement rounds in the full framework than in variants that remove recipe planning (w/o R) or both concept and recipe planning (w/o C+R).

Methodology in Plain English

The authors treat benchmark writing as a repeating build-test-fix loop, with an LLM agent acting as the builder.

A user names a scientific field. The framework first harvests "source priors" from two places: 100 randomly selected seed questions from an existing benchmark (CompBioBench, MatTools, or MedCTA), whose scientific domain, data modality, and required operations are extracted, and published scientific literature from the preceding five years, from which the agent extracts analytical workflows and generalizes them into reusable patterns.

From these priors the agent writes concepts — high-level statements of what capability should be tested. Each concept is then expanded into a recipe with seven parts, including where the data comes from (reuse existing benchmark data, retrieve from the web, or simulate), how the answer is established and concealed, how the answer is computed and verified, and how the question is phrased without revealing the solution steps. For clinical imaging, the agent is instructed to reuse existing benchmark images or retrieve real medical images rather than simulate them.

The recipe is then implemented into a concrete task: a question, a set of scientific data files, and a single ground-truth answer in a specified format so scoring can use exact match. Three independent solver trials are run per task on the question and files only. An LLM judge receives the question, data, reference answer, solver answers, and trajectories, then returns verifiability, difficulty (easy if all valid trials succeed, moderate if some, hard if none), and reasoning. If the task is not both verifiable and hard, the agent revises the recipe — keeping the concept — to force more validation, interpretation, or evidence integration. After four consecutive recipe revisions without success, it revises the concept itself and plans a new recipe.

Each concept is refined over up to 10 rounds including the initial construction round. When refinement ends, the agent reviews the records and writes an experience card. These cards form a memory that is fed into later concept generation. Concepts are generated in batches of five, with previously proposed concepts supplied to discourage duplicates.

For the reported experiments, Claude Code CLI with Claude Opus 4.8 and Codex CLI with GPT-5.6 Sol serve as generators, and each generator also serves as the solver during its own refinement. The paper constructs 50 tasks per domain through five experience-guided concept-generation iterations of 10 concepts each, and selects one task per refinement run for the final benchmark, prioritizing higher difficulty ratings.

Why This Matters

Impact on research. Benchmark construction in scientific domains depends on costly expert annotation, and the paper argues that this makes it hard to update evaluations as agents improve. If benchmarks can be generated and then revised using solver behavior, evaluation stop being a fixed artifact and starts co-evolving with the agents it measures — while the paper also documents how fast a newly generated benchmark saturates once a newer model appears.

Real-world applications.

• Drug discovery pipelines, where the paper explicitly notes that reliable agent analyses are critical for research.
• Genomics and transcriptomics analysis, illustrated by the UMI error-correction task where agents must separate sequencing artifacts from true molecular counts.
• Materials property calculation and tool use, where generated tasks expose differences among agents that the human-curated benchmark hides near its accuracy ceiling.
• Clinical imaging and multimodal clinical reasoning, where generated tasks reuse real medical images and are judged on verifiability, usefulness, and fitness.

Industry relevance. The work is conducted largely at Genentech (with KAIST co-authorship; the lead author's contribution was done during an internship at Genentech), which signals direct interest from a biopharmaceutical company in evaluating agents on internal-style scientific analysis. The framework's value proposition is reducing the labor and domain-expertise cost of maintaining evaluation suites and keeping them aligned with fast-moving model releases.

Future Directions

• Sustaining difficulty as agents improve. The paper shows that newer models erase the challenge of generated benchmarks (Opus 5.0 over Opus 4.8 by 19.5 percentage points; GPT-6 Astra over GPT-5.6 Sol by 15.1 points). Whether continued cycles of refinement and experience accumulation can outpace model progress, or whether the process itself saturates, is left open.

• Broadening beyond the three studied domains. Results cover computational biology, materials science, and clinical imaging, all seeded from specific existing benchmarks. Whether the concept/recipe abstraction generalizes to fields with different data modalities and verification norms is not established.

• Judge reliability. Verifiability and difficulty ratings come from LLM judges (Claude Opus 4.8 and GPT-5.6 Sol). The paper reports evaluator-specific results and inter-judge agreement in Appendix B.1, but the dependence of the whole refinement loop on judge quality remains a structural question.

• Separating generation capability from solving capability. The finding that a weaker solver (Claude Opus 4.8) produced harder benchmarks than a stronger solver (GPT-5.6 Sol) is documented but not explained; understanding what makes a model a good task author rather than a good task solver is an open question.

Target Audience

Researchers and engineers working on agent evaluation, benchmark design, or scientific AI tooling, particularly those in industry labs who must keep evaluation suites current against frequent model releases. It is also relevant to domain scientists in computational biology, materials science, and clinical imaging who consume agent benchmarks and need to know whether generated tasks are trustworthy. Readers without background in LLM agents and benchmark methodology will find the experimental sections dense.

Authors’ abstract

As agents rapidly evolve, existing benchmarks can become saturated, limiting their ability to distinguish capabilities and reveal remaining failure modes. Particularly in scientific domains, constructing and updating benchmarks requires substantial time, labor, and domain expertise, making it difficult to keep evaluation aligned with advances in agent capabilities. We address this challenge by investigating whether scientific-agent benchmarks can be automatically generated and iteratively adapted as agent capabilities evolve. We introduce AutoSciBench, a framework that represents each task as a high-level concept specifying the scientific domain, data modality, and required reasoning approach, together with a low-level recipe specifying how the question, environment, and ground-truth answer are constructed and verified. Agents attempt to solve each task, producing solver trajectories and corresponding judge feedback which AutoSciBench uses to revise the recipe or concept, closing observed shortcuts and shifting tasks toward raw-data re-examination, interpretation of intermediate results, and evidence integration. Experience distilled from completed refinement trajectories further guides new concept generation, allowing lessons from earlier task refinement to inform subsequent benchmark construction. Starting from existing benchmarks, we evaluate AutoSciBench across computational biology, materials science, and clinical imaging. Generated benchmarks reduce average solver accuracy by 22.4 and 25.5 percentage points relative to the human-curated benchmarks in computational biology and materials science, respectively, while generated tasks receive higher average quality ratings across all three domains, suggesting that scientific-agent evaluation can adapt as agent capabilities advance.