Evolving Agent Interfaces

AI summary

Overview

Research area: AI agents — specifically computer-use agents (CUAs), agent–computer interface design, and population-based evolutionary search over software artifacts.

Technical level: Advanced. The paper assumes familiarity with agent harnesses, tool-use APIs, LLM benchmark evaluation, and evolutionary algorithms, though the core idea is explained in accessible terms.

Scope: The paper proposes Agent S4, a meta-agent system that automatically constructs and evolves repositories of interfaces and usage knowledge for a fixed computer-use agent, and evaluates the resulting repositories on OSWorld V2 and ScienceBoard.

What This Paper Is About

Computer-use agents are constrained by interfaces that humans design and freeze before deployment, so an agent may click through a spreadsheet cell by cell when a single line of code would work. The authors argue that interface design should itself be an agent capability, and they build a system that studies an agent's own task trajectories and evolves new interfaces plus knowledge about when to use them. The goal is a repository of interfaces that improves task success for a fixed model and harness, without human redesign.

Key Contributions

1. Evolving Agent Interfaces from synthetic, verifier-free experience. Agent S4 evolves a repository of interfaces and usage knowledge from synthetic task trajectories that carry no success labels and no task verifiers, while the task agent's model and harness remain fixed. Validation tasks that the meta-agent never observes decide which repositories are developed further.

2. Evolutionary procedure over repositories. The paper formalizes interface evolution as producing a revised repository from a repository and its trajectories, and instantiates it with population-based evolution using binary tournament selection, mutation (one parent) and crossover (two distinct parents), with descendants retained alongside parents in a lineage DAG.

3. Empirical demonstration that evolved interfaces outperform fixed ones. With Qwen3.8-27B, the evolved repository reaches 21.30% binary success on OSWorld V2 versus 5.56% for the same model with fixed Qwen CUA interfaces, exceeding GPT-5.5 (13.00%) and Claude Opus 4.8 (20.60%) at a small fraction of their per-task cost; on ScienceBoard the evolved repository reaches 58.58% versus 41.4% for Qwen CUA.

4. Analysis of evolved interfaces and evolution strategies. The authors taxonomize evolved interfaces as System-wide, App-specific, Cross-App, and Task-specific, count new interfaces and extensions across runs, compare population versus linear evolution with and without a scratch VM for the meta-agent, and show that repeated validation separates consistently good repositories from ones with a lucky first score.

Main Findings

• Best OSWorld V2 binary success comes without a scratch VM. Population evolution without a scratch VM reaches 21.30% binary, 46.90% partial, and $1.10 per task. With a scratch VM, population evolution reaches 19.44% binary but a higher partial score of 51.06% at $1.27 per task.

• The same model improves dramatically through interfaces alone. The Qwen CUA baseline with Qwen3.8-27B and fixed interfaces scores 5.56% binary, 20.69% partial, at $1.39 per task. The evolved repository raises this to 21.30% binary and 46.90% partial while lowering cost to $1.10. The abstract describes this as 3.8 times the same model with fixed interfaces.

• Cost efficiency versus frontier models. GPT-5.5 (batched) scores 13.00% binary, 49.50% partial at $25.50 per task; Claude Opus 4.8 (batched) scores 20.60% binary, 54.80% partial at $72.40 per task. The evolved-repository result is described as about 4.3% and 1.5% of their respective costs.

• A seed repository alone already helps. On OSWorld V2, no repository gives 15.74% binary and 44.93% partial at $1.36 per task; the seed repository gives 19.44% binary and 44.90% partial at $1.11, and evolution then raises binary to 21.30% and partial to 46.90% at $1.10.

• Evolution strategy matters differently for binary versus partial score. Linear evolution with a scratch VM reaches 19.44% binary and 47.17% partial. Population with a scratch VM matches binary (19.44%) and raises partial to 51.06%. Population without a scratch VM gives the highest binary (21.30%) but lower partial (46.90%).

• Generalization to scientific software. On ScienceBoard's 169 tasks across six applications, Qwen CUA scores 41.4%, the seed repository 55.03%, and the evolved repository 58.58%, solving 99 tasks versus 93 and reducing average model calls per task from 61.7 to 58.9. The selected repository grows from 15 functions in the seed to 64. Gains are largest on KAlgebra, rising from 48% to 68%.

• Evolved interfaces widen what the agent can observe and do. On OSWorld V2 without a scratch VM, evolution introduces 4 new interfaces and 127 extensions, including 29 System-wide, 63 App-specific, and 40 Cross-App extensions. With a scratch VM, the population introduces 12 new interfaces and 253 extensions, with Cross-App extensions rising to 129; nine new interfaces are App-specific and nine Cross-App, with seven carrying both labels. On ScienceBoard, evolution introduces 7 new interfaces and 91 extensions, of which six new interfaces and 77 extensions are App-specific, plus three Task-specific extensions.

• Repeated validation corrects for run-to-run variance. On ScienceBoard, repositories if-003 and if-004 reach the highest first score of 73.33% at steps 3 and 4, but a second evaluation lowers their means to 60.00% and 66.67%. The selected repository if-016 scores 73.33%, 66.67%, 73.33%, and 73.33% across four evaluations for a mean of 71.67%.

• Qualitative case study. On OSWorld V2 task 008, the fixed-interface Qwen CUA agent opens receipt emails one at a time, handles five browser download warnings, twice tries to open an invoice before download completes, and needs eight scrolls and four clicks on "Load more" to find the $500 NeurIPS charge. The evolved agent uses browser.api to retrieve mailbox contents and attachment URLs as structured data, downloads three PDFs with curl, reads them with pdftotext, and reuses the browser interface to query the bank dashboard.

• Fixed interfaces already push agents toward code, but through the GUI. The paper cites that terminal commands account for 24.7% of fixed-interface agent actions and that GPT-5.5 devotes 78% of its budget to code, API, or file operations, with each such program typed through the desktop and read back from a screenshot.

Methodology in Plain English

The authors keep the task agent fixed: the same model (Qwen3.8-27B) runs in the same harness (a modified Prime Agent harness executing code in a persistent IPython kernel, with Medium reasoning, a 131,072-token context window, a 32,000-token output limit, and automatic context compaction). Only the repository of interfaces and knowledge changes.

The loop works like this. A population starts from one seed repository. At each step, an operator schedule picks mutation or crossover, and parents are chosen by binary tournament on validation score. A separate meta-agent, with its own computer holding the parent repositories, their training trajectories, and a lineage notebook, reads the trajectories and diagnoses failures and inefficiencies. It then writes code — adding, revising, or removing interfaces and usage knowledge — to produce one descendant repository.

That descendant is frozen and given to the task agent, which attempts 30 synthetic training tasks (48 for ScienceBoard) and 15 held-out validation tasks. Training attempts have no success labels; the meta-agent only sees those training trajectories. Validation tasks, trajectories, diagnostics, and scores stay hidden from the meta-agent, so validation affects construction only through which repositories get selected as parents. Every descendant joins the population alongside its parents regardless of score, preserving alternative designs for later recombination; after 10 steps the five lowest-scoring variants are removed from parent eligibility. Each run uses 30 descendants.

For the synthetic tasks, a coordinator (GPT-6 Astra), authors (GPT-5.6-Sol), and reviewers (GPT-5.6-Sol) produced independently authored scenarios that follow each benchmark's methodology without overlapping official tasks; no step graded agent attempts or produced reference solutions. A scratch-VM variant gives the meta-agent a machine with the same AMI as the benchmark tasks so it can open applications and test interface code during construction. Construction budgets are two hours wall-clock per construction on OSWorld V2 and one hour on ScienceBoard. Task rollouts use a 300-step budget on both benchmarks; new OSWorld V2 training and validation rollouts also get a two-hour wall-clock limit, ScienceBoard gets a 60-minute limit, and the final OSWorld V2 evaluation drops the wall-clock limit. Costs are estimated from token usage at Parasail's input and output rates assuming an 80% input-cache hit rate, excluding repository construction and runtime infrastructure.

Why This Matters

Impact on research. The paper reframes interface design as a learned, evolvable artifact rather than a fixed human decision, and shows gains that are complementary to model and harness improvements because the model and harness are held constant. It also offers a route to improvement that does not require success labels or task verifiers, which are expensive to build for open-ended computer use.

Real-world applications:

• Automating office work such as expense claims, spreadsheets, and document handling, where the evolved browser interface replaced repeated visual navigation and manual download handling.
• Scientific and technical software, where evolved interfaces for ChimeraX, GRASS GIS, KAlgebra, Lean, Celestia, and TeXstudio raised success on niche applications that general computer-use interfaces rarely cover.
• Reducing inference cost for agent deployments, since evolved interfaces let a 27B open-source model beat far larger and more expensive frontier models on binary task completion at a fraction of per-task cost.
• Agent framework and tooling development, where the output repository is itself reusable code that can be inspected, documented, and invoked.

Industry relevance. The result that a 27B open-source model at $1.10 per task exceeds GPT-5.5 at $25.50 and Claude Opus 4.8 at $72.40 in binary success on OSWorld V2 is a direct cost argument for investing in interface evolution rather than only in larger models. The code is open-sourced at the linked repository.

Future Directions

• Why does the scratch VM trade binary success for partial score? Population evolution with the scratch VM reached 19.44% binary and 51.06% partial, while without it the result was 21.30% binary and 46.90% partial. The paper attributes the higher partial score to additional application interfaces but does not resolve the binary trade-off.

• Can verifier-free evolution scale to more tasks and longer runs? All reported searches used 30 synthetic training tasks for OSWorld V2 and 48 for ScienceBoard, with 15 validation tasks and 30 descendants; whether larger training sets or longer evolution budgets keep improving repositories is not reported.

• How much does repeated validation generalize? Denoising was run on ScienceBoard every five steps but not on OSWorld V2, which the authors attribute to the much longer time and cost of evolution and repeated evaluations. Whether denoising also helps OSWorld V2 is left open.

• How far does evolved-interface transfer go? The paper runs a fresh evolution per benchmark from the same seed repository rather than testing whether a repository evolved on one domain transfers to another, and the related work section in the supplied content is truncated, so the authors' full positioning against prior interface and skill-composition methods is not reported here.

Target Audience

Researchers and engineers working on computer-use agents, agent harnesses, and tool-interface design; practitioners who deploy agents on desktop or scientific software and care about cost per completed task; and readers interested in evolutionary or search-based methods for improving LLM agent scaffolding. Readers focused on interface design in human–computer interaction will also find the interface taxonomy and the qualitative trajectory comparisons useful.

Authors’ abstract

Computer-use agents are limited by the interfaces humans give them. An agent may type a spreadsheet cell by cell when a single line of code would do, or print all two thousand lines of a file through a shell to find one function. Both the interfaces and the knowledge of when to use each are built by hand, which cannot keep pace with new applications, workflows, and models. We argue that interface design should itself be an agent capability. We introduce Agent S4, an autonomous computer-use scientist that builds Evolving Agent Interfaces from task experience. A meta-agent examines task agents' trajectories on synthetic training tasks, without success labels, and constructs and revises interfaces and usage knowledge in a repository. Population-based evolution then selects which repositories to develop further by their performance on validation tasks the meta-agent never sees, favoring changes that generalize beyond the seen tasks. On OSWorld V2, Agent S4, with Qwen3.8-27B and the evolved repository of interfaces, sets a new state of the art in open-source computer-use agents, reaching 21.3% binary score, 3.8 times the same model with fixed interfaces. This is above both GPT-5.5 (13.0%) and Claude Opus 4.8 (20.6%), at about 4.3% and 1.5% of their respective costs. On ScienceBoard, the evolved interfaces improve the success rate from 41.4% to 58.6% with the same model. These results suggest that useful interfaces can be evolved from experience, making autonomous interface design a source of improvement complementary to advances in models and harnesses. We open-source our code at https://github.com/simular-ai/agent-s.