AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

AI summary

Overview

• Research area: AI agent evaluation and benchmarking, specifically agents applied to scientific research tasks.
• Technical level: Advanced — the paper concerns agent harnesses, controlled experimental design, tool interfaces, and confounders such as model cost and tool access.
• Scope: The paper defines principles and tooling for rigorously benchmarking AI agents doing scientific research, and introduces AstaBench, a 2400+ problem suite spanning the scientific discovery process, with a controlled environment, baseline agents, and an evaluation of 57 agents across 22 agent classes.

What This Paper Is About

AI agents are increasingly proposed as tools for scientific work — automating literature reviews, replicating experiments, analyzing data, and suggesting new research directions — with both general "deep research" systems and science-specific agents such as AI Scientist and AIGS now in existence. The problem is that existing benchmarks cannot support rigorous comparison of these agents, which makes it hard to tell genuine capability advances from artifacts of setup, cost, or tool access. The paper's goal is to establish principles and tooling for more rigorous agent benchmarking, and to instantiate them in a holistic benchmark suite for scientific research assistance.

Key Contributions

1. A set of principles and supporting tooling for rigorous agent benchmarking, motivated by five specific shortcomings the authors identify in existing benchmarks: no reproducible agent tools for controlled comparison of core agentic capabilities; no accounting for confounders such as model cost and tool access; no standardized interfaces for quick agent prototyping and evaluation; no holistic, product-informed measures of real-world use cases like science research; and no comprehensive baseline agents needed to identify true advances.
2. AstaBench, a suite providing a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems that span the entire scientific discovery process and multiple scientific domains, including many problems inspired by actual user requests to deployed Asta agents.
3. A scientific research environment with production-grade search tools — described as the first of its kind — that enables controlled, reproducible evaluation and better accounts for confounders.
4. A comprehensive set of baseline agents, including nine science-optimized classes of Asta agents and numerous baselines, alongside an extensive evaluation of 57 agents across 22 agent classes.

Main Findings

• AI remains far from solving science research assistance: The abstract's stated headline result is that despite meaningful progress on certain individual aspects of the task, AI has not come close to solving the overall challenge of assisting with scientific research.
• Existing benchmarks are insufficient: The work is premised on the finding that current benchmarks fail along five distinct dimensions — reproducibility of agent tools, confounding variables, standardized interfaces, holistic real-world measures, and adequate baselines.
• Progress is uneven rather than uniform: The framing distinguishes "certain individual aspects" where progress is meaningful from the overall capability, implying that partial success does not translate into general competence at research assistance.
• Additional findings are not detailed in the abstract: The paper reports "several interesting findings" from evaluating 57 agents, but the abstract names only the most important one. No scores, per-task results, or comparisons between the 22 agent classes are given in the abstract, so the ranking of agents and the specific patterns behind these findings cannot be reported here.

Methodology in Plain English

The authors start by diagnosing why existing agent benchmarks are hard to trust, listing the five gaps above. From that diagnosis they derive principles for what a rigorous benchmark should do, then build tooling to satisfy them: a controlled scientific research environment with production-grade search tools, so that agents can be run under equivalent conditions and differences in cost or tool access do not silently distort comparisons. They then assemble a large problem suite — 2400+ problems covering the whole arc of scientific discovery across multiple domains — with many tasks drawn from real requests made to deployed Asta agents, which grounds the benchmark in actual use rather than synthetic exercises. To make the benchmark usable and to establish a meaningful reference point, they supply standardized interfaces for prototyping and evaluating agents, then build out a broad set of agents (nine science-optimized Asta agent classes plus many baselines) and run 57 agents spanning 22 agent classes through the suite.

Why This Matters

Impact on research: Benchmark quality determines whether reported agent progress is real. By addressing reproducibility, confounders, and baseline adequacy, this work aims to give researchers a comparison protocol in which a measured improvement reflects an actual capability gain rather than a difference in model cost, search access, or evaluation setup — a prerequisite for cumulative progress in the field.

Real-world applications:

• Literature review and synthesis: Agents that survey and summarize bodies of published work, one of the research activities the suite is designed to measure.
• Experiment replication: Reproducing published experiments, named as a target capability for these agents.
• Data analysis for researchers: Analyzing scientific data as part of the discovery workflow.
• Research ideation: Proposing new directions of inquiry, the most open-ended capability in the suite.
• Product evaluation of deployed agent systems: Because problems are inspired by actual requests to deployed Asta agents, the suite is described as providing product-informed measures usable for assessing systems in real use.

Industry relevance: Organizations building or deploying research agents — from general-purpose deep research products to specialized scientific agents — need comparable evaluation to justify claims of advancement. The standardized interfaces lower the cost of prototyping and evaluating new agents, and the explicit treatment of model cost as a confounder reflects the practical tradeoffs that commercial deployments face.

Future Directions

• Closing the capability gap: The central open question is what is required to move from progress on individual aspects of scientific work to reliable end-to-end research assistance, which the abstract states AI is far from achieving.
• Extending and maintaining the problem suite: The suite covers many domains and the full discovery process; coverage can be expanded and refreshed as new agent capabilities and new real user requests appear.
• Using the baselines to validate future claims: The broad set of Asta agent classes and baselines exists to help identify true advances, implying a continuing role in auditing new systems against them.
• Refining rigor around confounders: The paper's principles treat cost and tool access as confounders; further work concerns how thoroughly such variables can be controlled and reported as agent designs diversify.
• Comparative analysis across the 22 agent classes: The abstract reports an evaluation of 57 agents but does not present the comparisons, leaving the detailed breakdown of which agent classes do better or worse on which parts of the discovery process as material the full paper addresses.

Target Audience

Researchers and engineers working on AI agents, particularly those building or evaluating agents for scientific tasks; benchmark designers interested in methodology for reproducible, confounder-aware agent evaluation; teams behind general-purpose deep research systems and specialized scientific agents such as AI Scientist and AIGS who need external baselines; and research organizations or product groups assessing whether agentic systems are ready for real scientific workflows.

Authors’ abstract

AI agents hold the potential to revolutionize scientific productivity by automating literature reviews, replicating experiments, analyzing data, and even proposing new directions of inquiry; indeed, there are now many such agents, ranging from general-purpose "deep research" systems to specialized science-specific agents, such as AI Scientist and AIGS. Rigorous evaluation of these agents is critical for progress. Yet existing benchmarks fall short on several fronts: they often (1) lack reproducible agent tools necessary for a controlled comparison of core agentic capabilities; (2) do not account for confounding variables such as model cost and tool access; (3) do not provide standardized interfaces for quick agent prototyping and evaluation; (4) fail to provide holistic, product-informed measures of real-world use cases such as science research; and (5) lack comprehensive baseline agents necessary to identify true advances. In response, we define principles and tooling for more rigorously benchmarking agents. Using these, we present AstaBench, a suite that provides a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests to deployed Asta agents. Our suite comes with the first scientific research environment with production-grade search tools that enable controlled, reproducible evaluation, better accounting for confounders. Alongside, we provide a comprehensive suite of nine science-optimized classes of Asta agents and numerous baselines. Our extensive evaluation of 57 agents across 22 agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance.