Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools

AI summary

Overview

Research area: AI safety and biosecurity — specifically, benchmarking how LLM agents use biological AI models (BAIMs) in protein design workflows.

Technical level: Intermediate. The paper is readable without a biology background, though familiarity with LLM agents, tool use, and basic protein design concepts helps.

Scope: The paper introduces ABLE (Agentic BAIM–LLM Evaluation), a nine-task benchmark that scores 15 frontier language models on their ability to plan and execute a computational viral protein design workflow using tools such as ProteinMPNN and AlphaFold3, and compares them against 11 human experts on three of those tasks.

What This Paper Is About

Biosecurity researchers worry that LLM agents connected to protein design tools could make sophisticated protein engineering accessible to far more people, including those with malicious intent. Prior risk assessments evaluated LLMs and biological AI models (BAIMs) separately — for example, testing whether a chatbot will recite dangerous knowledge — but none tested whether an AI agent can actually operate protein design software end to end. ABLE fills that gap by decomposing a dual-use protein design workflow into individually scored steps and measuring both AI agents and human experts on how well they complete them.

Key Contributions

1. A new benchmark, ABLE, that breaks a computational protein design workflow into one planning task (ABLE0) and eight sequential execution tasks (ABLE1–8) spanning structure retrieval, structure preparation, critical residue identification, design strategy, tool selection, sequence generation with ProteinMPNN, validation tool selection, and variant ranking from AlphaFold3 outputs. Each task is scored 0 to 1 with partial credit, and each agent is run N = 10 times per task.

2. A large-scale model evaluation covering 15 frontier models, including the finding that seven closed-weight models (Claude Sonnet 4.5, Claude Opus 4, Claude Opus 4.1, Claude Opus 4.5, GPT-5, GPT-5.1, GPT-5.2) refused all tasks, while eight others completed at least some.

3. A human expert baseline from 11 recruited researchers with PhD-level or ≥5 years equivalent experience in computational protein design or structural biology, who each completed ABLE0, ABLE6, and ABLE8 under time limits of 1, 4, and 2 hours respectively, with AI assistants prohibited.

4. Task-level metrics intended for governance, proposed as inputs to capability thresholds, managed deployment decisions, and tiered API access policies for BAIM–LLM systems.

Main Findings

• Two models lead, but no model is uniformly reliable. Claude Sonnet 4 and Gemini 3 Pro achieved the highest overall scores. Both scored a perfect 1.00 on all 10 runs for finding the correct protein structure (ABLE1), fetching and preparing the structure file (ABLE2), and selecting appropriate design and validation tools (ABLE5 and ABLE7).
• Only Gemini 3 Pro achieved workflow success. The paper defines workflow success as a perfect score of 1.0 at least once on every execution task (ABLE1–8). Gemini 3 Pro was the only model to do so, but its success rates on ABLE3 and ABLE4 were each 30%, showing that reliable end-to-end completion remains inconsistent.
• Claude Sonnet 4 reached partial success everywhere. It scored ≥0.5 on all tasks but never achieved a perfect score on ABLE3 (critical residue identification), where its mean score was 0.48 ± 0.06.
• Open-weight models were weaker and more erratic. GPT-OSS 120B, Qwen3 235B, DeepSeek V3.2, and Kimi K2 all performed worse than the best closed-weight models on most tasks, particularly on long tool-use tasks (ABLE6 and ABLE8). Kimi K2 and DeepSeek V3.2 frequently terminated tasks early without calling any tools.
• One open-weight result looks like memorization. GPT-OSS 120B scored best of all models on ABLE3 (0.89 ± 0.05 mean, 0.5 success rate) but worst on ABLE6 and ABLE8 (0.00 mean on ABLE6, 0.05 ± 0.05 on ABLE8). Its transcripts showed little to no tool use for ABLE3, suggesting it recited from training data.
• Humans beat models at hands-on execution, lost at planning and interpretation. Human baseliners averaged 0.86 ± 0.09 on ABLE6 (variant generation) — matching or exceeding every frontier model — and 0.70 ± 0.08 on ABLE8. On ABLE0 (plan generation) they averaged 0.55 ± 0.07, versus 0.88 for Claude Sonnet 4, 0.88 for Gemini 3 Pro, and 0.92 for GPT-OSS 120B.
• Prior tool familiarity matters for humans. Participants with preferred qualifications (hands-on experience with AlphaFold, ProteinMPNN, ESM, or Rosetta) outperformed minimal-qualification participants on ABLE6 by 22 percentage points (0.94 vs. 0.72).
• Models understand tool categories well, but not fine-grained workflow details. All non-refusing models scored high on selecting appropriate design and validation BAIMs (ABLE5, ABLE7). On ABLE0, models reliably captured core workflow steps but scored substantially worse on criteria assessing details, and no model produced a procedure capturing every step and detail assessed in ABLE1–8.
• Failure modes were environmental, not conceptual. Models generally understood which metrics determine structural stability. They were mostly limited by interacting with the agent environment and calling tools correctly. Several models hallucinated tool outputs instead of calling tools; only Claude Sonnet 4 and Gemini 3 Pro entirely avoided this. Gemini 2.5 Pro, Grok 4, Kimi K2, and DeepSeek V3.2 frequently stopped early.
• Content filtering blocked a large share of the frontier. Seven closed-weight models from Anthropic and OpenAI refused every ABLE task, which the authors interpret as deliberate developer safety choices. GPT-OSS 120B refused on some runs but not others.

Methodology in Plain English

The researchers took a real computational protein design workflow and cut it into nine self-contained tasks. Each task hands the agent everything it needs — a natural-language prompt, the correct answers from earlier steps where relevant, and a set of tools — and asks for a single submission in a machine-parsable format. Some tasks only need web search (finding a structure, identifying critical residues, choosing tools); others require running Biopython, ProteinMPNN, or reading AlphaFold3 output files.

Scoring is automatic: eight of the nine tasks are graded by comparing answers against predefined expected values, reference files, or expert-curated lists of residues, mutations, and best sequences. Only ABLE0, the high-level plan generation task, uses a model-graded rubric built from 6 core completeness criteria and 7 detail criteria, each scored 0, 0.5, or 1. Reported ABLE0 scores in the main results table reflect the completeness criteria only.

The benchmark was implemented on the UK AI Security Institute's Inspect AI framework (MIT License), with every task run in an isolated container. ProteinMPNN was hosted on a t3.large EC2 instance, and models got a lightweight utility tool to run remote commands on it. Grading criteria were developed in consultation with biology experts.

For the human baseline, 11 qualified researchers completed ABLE0, ABLE6, and ABLE8 under time limits, using web search, a command line, the same EC2 instance, and AlphaFold3 output files. They were scored with the same criteria as the models. AI assistants were banned, and participants installed browser extensions blocking AI-generated search results. Sessions were run remotely, cloud instance activity was logged, and screen recording was optional. Participants also filled out a post-task survey on perceived difficulty, time pressure, and subjective success.

Why This Matters

Impact on research: ABLE is the first benchmark, according to the authors, that tests agentic tool use in a biosecurity-relevant protein design domain rather than testing knowledge or isolated models. It provides a reusable scaffold and task-level metrics that can track capability change as BAIMs and LLM agents improve, and it produces direct evidence that frontier models match or exceed domain experts on reasoning and interpretation steps of the workflow.

Real-world applications:

• Biosecurity and screening policy: ABLE-style evaluations could inform capability thresholds, managed deployment requirements, and mitigation testing (refusals, model unlearning, prompt filtering) before a model is released.
• Access control design: The authors suggest tiered API access based on verified research credentials to limit misuse while preserving legitimate research.
• Governance gap analysis: The paper notes that the 2024 U.S

Authors’ abstract

We introduce ABLE, a benchmark for evaluating LLM agents' ability to use biological AI models (BAIMs), such as ProteinMPNN and AlphaFold3, in dual-use protein design workflows. ABLE assesses agent performance through a set of tasks spanning structure retrieval, sequence generation, and design validation. We evaluate 15 frontier models and find that seven refuse all tasks, while the remaining models exhibit substantial performance differences. Claude Sonnet 4 and Gemini 3 Pro achieve the highest scores across information retrieval, tool selection, and tool use. We further compare model performance on a subset of tasks against an expert human baseline. Our results suggest that current LLMs can substantially lower barriers to protein design, but remain inconsistent in planning, strategy generation, and integrating biological knowledge with tool use.