BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics

AI summary

Overview

• Research area: AI agent evaluation; LLM agents for bioinformatics and computational biology workflows; benchmark design and robustness testing.
• Technical level: Intermediate. The paper assumes familiarity with LLM agent concepts (harnesses/scaffolds, tool use, transcripts) and with common bioinformatics pipeline stages, but its prose is largely self-defining and accessible to readers with a general ML or computational biology background.
• One-sentence scope: The paper introduces BioAgent Bench, a manually curated suite of ten end-to-end bioinformatics tasks with an LLM-based grader and controlled robustness perturbations, and uses it to compare frontier closed-weight and open-weight models running inside three agent harnesses.

What This Paper Is About

Bioinformatics analyses consist of long chains of command-line tools, heterogeneous file formats, reference datasets, and statistical decisions, which makes them hard to evaluate with simple pass/fail question-answering or code-generation benchmarks. The authors build an evaluation suite of realistic, end-to-end bioinformatics pipelines with concrete ground-truth outputs, then test whether frontier LLM agents can complete them and, more importantly, whether they remain reliable when inputs are corrupted, decoy files are present, or prompts are deliberately bloated. The goal is to make agent failure modes measurable and to accelerate the development of cost-effective, locally deployable agents for workflows involving sensitive patient data or unpublished intellectual property.

Key Contributions

1. A benchmark dataset of ten manually curated end-to-end bioinformatics tasks, spanning bulk and single-cell RNA-sequencing, comparative genomics, variant calling, metagenomics, viral metagenomics, transcript quantification, and experimental evolution, each with a natural-language prompt, primary input data, and reference data where available.
2. A systematic comparison of closed-weight and open-weight models in this setting, run across three agent harnesses (Claude Code, Codex CLI, OpenCode), with the explicit framing that results measure the combined model-plus-harness agentic capability rather than the model alone.
3. An evaluation suite that grades intermediate steps as well as final outputs, using an LLM-based grader (GPT-5.1) that reports steps_completed, steps_to_completion, final_result_reached, results_match, and an F1-score where applicable (only the giab task in the current framework).
4. A robustness methodology built on controlled perturbations — prompt bloat, synthetically corrupted inputs, and decoy files — combined with multi-trial stability metrics (Jaccard index for categorical outputs, Pearson correlation for numerical outputs).

Main Findings

• Frontier models complete pipelines at high rates. Claude Opus 4.5 attains a 100% completion rate, while Gemini 3 Pro, GPT-5.2, and Sonnet 4.5 each exceed 90%. The top results were obtained using the Codex CLI harness.
• Open-weight models trail. The best-performing open-weight model, GLM-4.7, reached 82.5% in Codex CLI, with other open-weight models ranging down to 65%.
• Planning quality correlates with completion. Plans scored by GPT-5.1 on a 1–5 scale correlated with overall agentic performance at Pearson R = 0.61, but the relationship is not deterministic — some models (e.g., Gemini-Pro-3) produced weaker plans yet still completed pipelines at high success rates.
• Results vary substantially across repeated runs. For GPT-5.2 in Codex CLI over four trials per task, the mean Jaccard overlap was 0.43 and the mean Pearson correlation coefficient was 0.73, indicating considerable trial-to-trial variability attributed to tool non-determinism and differing intermediate parameter or statistical choices.
• Corruption was detected in 7/10 tasks. Per-task outcomes are tabulated in Table 2; several failure modes emerged, including continuing analysis over corrupted inputs (alzheimer-mouse, comparative-genomics), flagging corruption but continuing anyway (evolution, giab, metagenomics, transcript-quant), and early termination either appropriately (single-cell) or due to cascading failures (deseq).
• Decoy files were used erroneously in 2/10 tasks. In comparative-genomics the agent globbed files by the .genomic.fna filename suffix and included the decoy organism; in metagenomics it selected a viral reference database instead of the required bacterial database.
• Prompt bloat degraded performance markedly. Agents completed 28% fewer steps relative to the unperturbed setting overall; several tasks showed −100% delta completion, where the agent restated the task, cycled through superficial reformulations, and terminated early without producing substantive intermediate artifacts.
• Pipeline completion overestimates reliability. The authors argue that correct high-level pipeline construction does not guarantee correct step-level reasoning, and that completion should be treated as a necessary but insufficient success criterion.
• Qualitative failure-mode differences. Manual inspection of a subset of runs suggested open-weight models were more prone to repeated error-correction loops or premature termination, whereas frontier models more often recovered and completed the pipeline.

Methodology in Plain English

The authors assembled ten bioinformatics tasks that they considered widely used and easy to express as tool-calling pipelines, deliberately steering the design toward software-engineering-style benchmarks rather than open-ended biology-analysis benchmarks, so the suite could support post-training uses such as reinforcement learning, distillation, or harness tweaks. Each task bundles a natural-language prompt with a specified output format and the files needed to run and grade it; reference implementations were written in Python (three tasks), R (two tasks), and bash (five tasks). Seven tasks require tool interaction and four are verifiable with a pass/fail label.

Two hard constraints shaped dataset selection: runtime must be under 4 hours and workflows must run within 48GB of RAM, which favors smaller organisms and directly provided reference data, and excludes large-organism workflows such as human sequencing as well as the work of finding and staging large external references.

For each trial the agent receives a system prompt, input files, and task instructions, and runs sandboxed in a hashed run directory tied to the evaluation, with internet access and "high" reasoning effort where available. Agents write stage-scoped artifacts under an outputs/ directory and final results under results/. Instead of hard-coded success criteria, an LLM grader (GPT-5.1) compares the agent's output and trace against a rubric and ground truth, which accommodates multiple valid solution paths (for example, GATK4 Haplotype caller versus DeepVariant for germline variant calling) and variable step counts. The grading logic prioritizes evidence of pipeline completion over numerical accuracy. Robustness was probed by running multiple trials, inflating prompts with topically related but non-essential text, synthetically corrupting selected input files, and introducing decoy files the agent should ignore. The primary metric is completion rate: the percentage of required steps the agent demonstrably completes and the production of the requested CSV or TSV artifact.

Why This Matters

Impact on research. The paper reframes agent evaluation from "Can it produce correct outputs?" to "Can it produce correct outputs reliably, for the right reasons, while making correct decisions throughout the pipeline?" It makes step-level brittleness — shallow filename heuristics, weak input validation, sensitivity to distraction — into something measurable, and it provides an openly released suite (github/bioagent-bench and github/bioagent-experiments) intended as a target for fine-tuning, distillation, and reinforcement learning on verifiable multi-step tasks.

Real-world applications:

• Automating routine bioinformatics workflows — alignment, quantification, variant calling, metagenomic classification — as workflow assistants that reduce manual overhead.
• Clinical and diagnostic settings where the crucial question is whether an agent can detect when it should not proceed and can justify its choices with data-grounded evidence.
• Privacy-constrained environments such as tumor sequencing or clinical metadata that cannot be sent to third-party model providers, motivating locally deployable open-weight agents.
• Protection of proprietary reference collections and unpublished intellectual property held inside an institution's secure environment.

Industry relevance. Model and harness developers get a task suite for differentiating agentic competence from domain knowledge; research and clinical organizations get a standardized way to track the capability gap between open-weight and closed frontier models before deploying automation on sensitive data.

Future Directions

• Expand task and dataset diversity, including larger and messier inputs, and add tasks that require sourcing and justifying external references and best practices from primary sources — an area the current suite explicitly excludes.
• Integrate robustness into the primary metrics, rather than reporting it as a separate, single-trial-per-condition analysis.
• Strengthen robustness evaluation with richer perturbations and automated scoring, covering confounders such as multiple plausible files, misleading names or metadata, partial truncations, and subtle format violations that remain syntactically valid but biologically implausible.
• Systematically characterize failure modes, including the repeated error-correction loops and premature termination behaviors that manual inspection only suggested qualitatively, and investigate why open-weight models underperform — weaker end-to-end agentic competence versus weaker explicit planning.

Target Audience

Researchers working on LLM agents and agent evaluation; bioinformatics and computational biology practitioners interested in automating pipelines; model and harness developers focused on tool use, error recovery, and long-horizon state tracking; and groups in clinical, pharmaceutical, or genomics settings who need to assess whether locally deployable open-weight agents can meet privacy, reliability, and governance requirements. Readers should note the paper's stated limitations: LLM grading can be subjective and biased by rubric wording and trace verbosity, the runtime (<4 hours) and memory (≤48GB RAM) caps reduce real-world fidelity, and the perturbation analysis uses a single trial per task and condition, so those results should be treated as suggestive.

Authors’ abstract

We introduce BioAgent Bench, an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The suite consists of manually curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) accompanied by task-specific prompts and concrete output artifacts to support automated assessment. We evaluate frontier closed- and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that agents based on frontier LLMs can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. By releasing the code and the complementary resources constituting our suite, our primary goal is to accelerate the development of cost-effective yet reliable local agents, capable of handling complex bioinformatics workflows often involving sensitive patient data or unpublished intellectual property.