AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts

AI summary

Overview

Research area: Evaluation of autonomous LLM-based agents — specifically, benchmarking long-horizon, real-world agentic capability.

Technical level: Advanced (assumes familiarity with LLM agents, tool-use scaffolds, rubric-based evaluation, and LLM-as-judge methodology).

Scope: The paper introduces AgencyBench, a benchmark of 138 tasks across 32 real-world scenarios in 1M-token contexts, plus a fully automated user-simulation and Docker-sandbox evaluation pipeline used to compare proprietary and open-source frontier models.

What This Paper Is About

Existing agent benchmarks tend to test a single capability (tool use, software engineering, or research) and lack the long-horizon depth of real work, while realistic tasks often require human-in-the-loop feedback that cannot be scaled. The authors build AgencyBench to test six core agentic capabilities across 32 scenarios and 138 tasks that average 90 tool calls, 1 million tokens, and hours of execution time per scenario. They pair it with a user simulation agent for iterative feedback and a Docker-based sandbox for automated, rubric-based scoring, so that rollouts can be collected and graded without human intervention.

Key Contributions

1. A challenging long-horizon benchmark. 138 authentic tasks across 32 scenarios, organized under 6 agentic capabilities (game development, front-end development, back-end development, code generation, research, and MCP tool use), each task defined by a query, deliverables, and rubrics.
2. A unified automated evaluation framework. A user simulation agent (Claude-4-Sonnet, temperature 0.0) supplies iterative feedback in an isolated workspace, deliverables are synced to a Docker-based remote sandbox that emulates human-computer operations (UI rendering, mouse/keyboard inputs), and artifacts are transferred to a separate eval-space for rule-based or LLM-as-judge scoring on a 0–10 scale.
3. A comprehensive evaluation of frontier models. Scores are reported for closed-source models (GPT-5.2, Claude-4.5-Opus, Claude-4.5-Sonnet, Gemini-3-Pro, Grok-4.1-Fast) and open-source models (GLM-4.6, Kimi-K2-Thinking, DeepSeek-V3.2, Qwen-3-235B-A22B-Thinking), quantifying the proprietary/open-source gap and behavioral differences.
4. An analysis of agentic scaffolds. An ablation over 10 representative scenarios compares the authors' custom scaffold against the Claude-Agent-SDK and OpenAI-Agents-SDK, revealing a "home-field advantage" for proprietary models in their native ecosystems.

Main Findings

• Proprietary models lead by a wide margin. Closed-source models average 48.4% versus 32.1% for open-source models. Within closed-source, scores range from 56.5% (GPT-5.2) down to 44.3% (Grok-4.1-Fast); within open-source, from 38.6% (GLM-4.6) down to 27.0% (Qwen-3-235B-A22B-Thinking).
• Capability specialization is uneven. Gemini-3-Pro leads on game (60.7%) and front-end (81.0%). GPT-5.2 leads on back-end (61.0%) and code (50.7%). Claude-4.5-Sonnet scores highest on research (71.4%). Grok-4.1-Fast scores highest on MCP (68.6%).
• More attempts are not equally useful to all models. Pass@1 rose to Pass@2 by 88.9% for GPT-5.2, 85.7% for Claude-4.5-Sonnet, 80.0% for Claude-4.5-Opus, 33.3% for Gemini-3-Pro and GLM-4.6, 25.0% for Grok-4.1-Fast, 300.0% for Kimi-K2-Thinking, 199.7% for Qwen-3-235B-A22B-Thinking, and 0.0% for DeepSeek-V3.2, which the authors say adheres to erroneous paths despite external critique.
• Self-correction is a distinct axis from initial quality. Gemini-3-Pro matches GPT-5.2's Pass@1 (28.1) but gains far less from feedback (33.3% Rise vs. 88.9%).
• Resource profiles differ sharply. GPT-5.2 consumes 3.4M tokens and 89 turns; Claude-4.5-Sonnet 4.1M tokens, 0.9h, 64 turns; Gemini-3-Pro 1.8M tokens, 0.3h, 37 turns; Grok-4.1-Fast 1.2M tokens, 0.3h, 37 turns; Claude-4.5-Opus 1.7M tokens, 0.8h, 36 turns; GLM-4.6 2.4M tokens, 0.6h, 105 turns; Kimi-K2-Thinking 2.8M tokens, 1.2h, 65 turns; DeepSeek-V3.2 1.5M tokens, 1.0h, 21 turns; Qwen-3-235B-A22B-Thinking 1.2M tokens, 1.4h, 21 turns.
• Efficiency has different winners than raw score. GPT-5.2 has the highest attempt efficiency (38.7%), Grok-4.1-Fast the highest token efficiency (37.2%), and Claude-4.5-Sonnet the lowest token efficiency (11.4%), with its 4.1M tokens not yielding proportional gains.
• Models exhibit distinct tool-use "personalities." Claude-4.5-Opus and GPT-5.2 favor shell execution (45.5% and 43.5%); Gemini-3-Pro is the only model with substantial memory-tool use (6.9%); Qwen-3-235B-A22B-Thinking relies extremely on file operations (77.6%); Grok-4.1-Fast and GLM-4.6 lean on web search (9.5% and 8.6%). In raw counts, GPT-5.2 issues 425 run_shell_command calls, GLM-4.6 issues 158 list_directory calls and 381 write_file calls, and GPT-5.2 issues 146 replace (precision-edit) calls.
• Scaffold choice changes scores substantially. On 10 representative scenarios, Claude-4.5-Opus improves by 20.5% on the Claude-Agent-SDK (50.8 to 71.3); GPT-5.2 scores 58.7 on the OpenAI-Agents-SDK versus 57.4 on the authors' scaffold; GLM-4.6 improves 10.6% and Minimax-M2 8.6% on the Claude-Agent-SDK; Kimi-K2-Thinking degrades by 12.8% on the OpenAI-Agents-SDK.
• Evaluation reliability was validated. The user simulation agent achieved an average human-agreement score of 4.69 on 50 sampled rollouts; the LLM judges achieved a Kappa score of 0.93 against human annotations on a held-out set of 50 tasks.

Methodology in Plain English

The authors had 20 human experts — AI researchers, AI practitioners, and software engineer developers — build the benchmark. Each of the 32 scenarios contains 1 to 5 tasks in ascending difficulty, presented in sequence so earlier results affect later ones, totaling 138 tasks. For every task, experts wrote and verified a query, a set of deliverables, rubrics, and an executable evaluation script; a separate panel of four experts reviewed the whole dataset, and a task was only finalized under a unanimous-consensus policy requiring 100% approval.

At evaluation time, each task runs in an isolated workspace with an agent scaffold (file manipulation, command-line execution, web search, context management, and other tools). The agent interacts multi-turn to produce deliverables. A user simulation agent checks deliverables against the rubrics and, if the task does not clear the 60% score threshold, returns the failed rubrics and reasons for failure, prompting another attempt (up to K rounds). The user simulation agent is Claude-4-Sonnet at temperature 0.0.

Deliverables are then synced to a Docker-based remote sandbox that emulates human-computer operations — rendering UIs, clicking, screen recording, and capturing screenshots — to produce visual artifacts. These artifacts and raw deliverables are moved to a separate eval-space and scored 0–10 by executable scripts. Objectively verifiable tasks use rule-based assertion logic; subjective or visual tasks use an LLM-as-judge, with a text-based judge (Claude-4-Sonnet, temperature 0.0) for code quality and logic and a vision-based judge (Gemini-2.5-pro, temperature 0.0) for dynamic behavior and visual correctness. When both judges apply, the final score is their average.

Models were all accessed via the OpenRouter API with temperature 0.7. Reported metrics are Average Score (S_Avg, the rubric percentage), Average Attempts (Att, mean feedback rounds used per task), Pass@k (share of tasks meeting the 60% threshold within k feedback rounds, reported for k=1 and k=2), and two efficiency ratios: attempt efficiency (S_avg divided by Att) and token efficiency (S_avg divided by average tokens per scenario).

Why This Matters

Impact on research: The paper argues that agentic performance is not purely a model-intrinsic property but a coupling between model and scaffold, and that existing benchmarks lack the tool-call depth to differentiate frontier models. AgencyBench is positioned as a diagnostic testbed rather than only a leaderboard, and the full benchmark and evaluation toolkit are released at https://github.com/GAIR-NLP/AgencyBench.

Real-world applications:

• Selecting models and scaffolds for long-horizon autonomous workflows where tasks span millions of tokens.
• Diagnosing why an agent fails — e.g., poor self-correction, weak context persistence, or inefficient tool choice.
• Budgeting compute and API spend, using the attempt-efficiency and token-efficiency metrics.
• Validating full-stack deliverables such as games, front-end interfaces, back-end services, and MCP integrations in an isolated sandbox.

Industry relevance: The authors frame the work around economic production, and the resource-consumption results (tokens, wall-clock hours, and turns per scenario) speak directly to the cost of deploying agents. The scaffold-ablation results matter commercially because proprietary SDKs can move scores by double-digit percentages.

Future Directions

• Resource-efficient autonomy. The authors note that even the most advanced models struggle to master long-horizon autonomy without substantial resource consumption, and call for more resource-efficient agents.
• Scaffold-agnostic agents. Because models show a "native advantage" in their own ecosystems, a stated goal is agents whose performance does not depend on a specific agentic framework.
• Stronger self-correction. The wide spread in feedback gains — from 0.0% for DeepSeek-V3.2 to 300.0% for Kimi-K2-Thinking — points to self-correction as an open capability to improve.
• Broader coverage. The limitations section notes that the benchmark is confined to software-based agents within a computer interface and does not cover embodied agents or physical-world interaction, and that rapid model releases mean the evaluation is a snapshot that cannot exhaustively cover every model variant or checkpoint.

Target Audience

Researchers and engineers working on LLM agents, agentic scaffolds, and tool-use evaluation; benchmark designers needing an automated, rubric-based pipeline that avoids human-in-the-loop bottlenecks; and practitioners choosing models, frameworks, or compute budgets for long-horizon autonomous deployments.

Authors’ abstract

Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. However, existing benchmarks remain focused on single agentic capability, failing to capture long-horizon real-world scenarios. Moreover, the reliance on human-in-the-loop feedback for realistic tasks creates a scalability bottleneck, hindering automated rollout collection and evaluation. To bridge this gap, we introduce AgencyBench, a comprehensive benchmark derived from daily AI usage, evaluating 6 core agentic capabilities across 32 real-world scenarios, comprising 138 tasks with specific queries, deliverables, and rubrics. These scenarios require an average of 90 tool calls, 1 million tokens, and hours of execution time to resolve. To enable automated evaluation, we employ a user simulation agent to provide iterative feedback, and a Docker sandbox to conduct visual and functional rubric-based assessment. Experiments reveal that closed-source models significantly outperform open-source models (48.4% vs 32.1%). Further analysis reveals significant disparities across models in resource efficiency, feedback-driven self-correction, and specific tool-use preferences. Finally, we investigate the impact of agentic scaffolds, observing that proprietary models demonstrate superior performance within their native ecosystems (e.g., Claude-4.5-Opus via Claude-Agent-SDK), while open-source models exhibit distinct performance peaks, suggesting potential optimization for specific execution frameworks. AgencyBench serves as a critical testbed for next-generation agents, highlighting the necessity of co-optimizing model architecture with agentic frameworks. We believe this work sheds light on the future direction of autonomous agents, and we release the full benchmark and evaluation toolkit at https://github.com/GAIR-NLP/AgencyBench.