EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?

AI summary

Overview

• Research area: LLM-based agentic systems — specifically the evaluation of tool-using, skill-retrieving, and multi-agent systems under continually changing capability sets.
• Technical level: Intermediate. No new model architecture is proposed; the work is a benchmark and empirical study, but it assumes familiarity with agent harnesses, continual learning concepts (forward/backward transfer), and multi-agent coordination.
• Scope: A benchmark and empirical study showing that state-of-the-art LLM agents fail to maintain performance when the externally supplied "harness" of tools, skills, and specialist agents expands over time.

What This Paper Is About

LLM agents do not act alone — they act through a harness: the tools they can call, the reusable skills they can retrieve, and the specialist agents they can delegate to. In real deployments this harness is never static; new tools, skills, and agents are added continuously. The paper asks a simple but under-studied question: when the harness expands, can agents keep doing what they already could do, and can they learn to exploit the new capabilities? EvoHarnessBench is built to answer both halves of that question systematically.

Key Contributions

1. A new source of non-stationarity. The paper is the first benchmark to place non-stationarity in the externally supplied harness rather than in the task stream or in the agent's own self-generated capabilities. Prior continual-learning benchmarks for agents keep the harness fixed or evolve the tasks; EvoHarnessBench evolves the tools, skills, or agents available to the agent while tasks stay fixed once introduced.

2. A deterministic construction pipeline over three axes. The benchmark converts existing verifier-based agent benchmarks into nested, cumulative harness streams along three independent axes — tools (expanding API catalogs), skills (growing libraries of procedural know-how), and agents (expanding pools of entity-scoped specialists). Capabilities are released in frequency order (core first, long tail later), each task is assigned to the earliest stage at which all its required capabilities exist, and every task is guaranteed to need at least one capability introduced at that stage.

3. Two complementary evaluation modes. Deployment evaluation disables all cross-stage persistence and isolates the pure effect of harness growth. Self-evolving adaptation evaluation enables persistent state (memories, prompts, or learned code) that carries across stages, testing whether experience accumulated under earlier, narrower harnesses remains useful as new capabilities appear.

4. A benchmark of 17 harness streams. Each stream has 3–6 cumulative stages, drawn from EnterpriseOps-Gym (stateful enterprise workflows) and Agentic Last Exam (diverse software-tool tasks). In total: 802 unique tasks, 1,510 axis-specific evaluation examples, 520 executable tools, 42 latent reference skills, and 62 specialist agents — plus task-specific reference conditions for controlled comparison.

Main Findings

• Harness-induced forgetting is real and large. Simply expanding the harness degrades performance on tasks the agent previously solved, even though the required capabilities are still present and the model weights are unchanged. Deployment drops of 12.1% (tools), 13.8% (skills), and 46.4% (agents) are reported. The capabilities still exist; they are buried in a larger pool of alternatives.

• Broader tool exposure is a double-edged sword. Exposing frontier agents to the full cumulative tool catalog improves pass rate on both EOG and ALE relative to task-specific tools, but at substantially higher token cost. The larger catalog is not merely a distractor — it offers usable affordances — but exploiting them requires far more search and execution.

• Self-evolving adaptation gains are inconsistent. On the tool axis, MemToolAgent reaches 38.6%, ReasoningBank 36.9%, and Meta-Harness 35.2% on EOG versus a 30.2% deployment baseline; on ALE, most methods stay at or below baseline. Gains vary by method, environment, and stage.

• Retention and adaptation can be in tension. Methods that best preserve earlier competence (GEPA, Meta-Harness achieve positive backward transfer) show sharply negative forward transfer on ALE — down to −28.5% and −11.1% respectively — meaning adaptation over-specializes to observed experience and interferes with generalizing to held-out tasks.

• Skills behave differently from tools. Expanding the skill library alone has almost no effect on deployment performance (Codex stays at 18.9% on EOG), likely because skills must be explicitly retrieved rather than continuously exposed. The bottleneck is not execution under a broad library but learning when and how to retrieve relevant skills. Skill engagement is sparse and model-dependent: default GPT-5-based systems invoke almost none of the offered skills, while a task-specific GPT-5.5 invokes 82% and Claude Code 15%; task-specific GEPA raises GPT-5's invocation rate to 31%.

• Adaptation context matters. Restricting adaptation to task-specific tools leaves MemToolAgent's final performance nearly unchanged but cuts adaptation time from 23.9h to 17.9h; restricting skills lowers GEPA's pass rate slightly (24.1% → 23.0%) but improves score (61.0 → 63.1) and shortens adaptation from 4.1h to 2.9h. The main difficulty lies in adapting under a broad harness, not in the stage-wise evolution itself.

• Agent-pool expansion is environment-dependent. Unlike tools and skills, expanding the specialist-agent pool produces no consistent trend. Even with the task-specific reference pool, performance is low (multi-agent coordination is hard). Moving to the cumulative pool improves Codex on EOG (6.5% → 8.8%) but degrades it on ALE (5.3% → 4.2%). On EOG, however, this axis yields the largest relative gains from self-evolving adaptation — Meta-Harness reaches 18.5% from an 8.8% deployment baseline.

• Memory design matters more than memory presence. For single agents, structured memories beat raw trajectory replay, but adding explicit tool schemas to ReasoningBank hurts (down to 30.3%), and success-only vs. failure-only trajectories perform similarly (34.4%). For multi-agent systems, concrete trajectories can narrow search in AutoGen while abstract or broadly shared memories increase exploration, redundancy, and cost.

• Sequential skill learning can be harder than joint learning. The strongest skill-learning method, Batch Teacher Feedback, performs better when the same experience is learned jointly (23.6%) than when accumulated sequentially across stages (22.1%). Naive skill generation can perform worse than having no learned skills at all.

Methodology in Plain English

The authors take two existing agent benchmarks where tasks have verifiable outcomes (EnterpriseOps-Gym and Agentic Last Exam) and mechanically re-slice them into staged "release schedules."

For each axis:

• Tools: The source benchmarks already annotate which tools each task needs. Tools are counted by how many tasks use them, sorted by frequency, and partitioned into release buckets — common tools first, rare tools later. Stage t exposes every tool released up to t.
• Skills: System prompts in the source data mix general instructions with reusable step-by-step procedures. The authors strip out the procedural parts using rules (not an LLM) to form a hidden skill library, then match each task to skills by checking whether entities in the verifier-checked final state appear in a skill's procedure text. This is explicitly a heuristic proxy, not a ground-truth necessity annotation. At evaluation time the procedural text is removed from the prompt, so agents must retrieve skills from the pool.
• Agents: Each tool is mapped to the entity it operates on (a database, a filesystem, an email account). Tools sharing an owner form a bundle, and each bundle becomes an entity-scoped specialist agent. A lead agent with no direct tool access must delegate. Tasks spanning multiple entities structurally require multi-agent coordination.

Tasks are then assigned to the earliest stage where all their required capabilities are available, with the added constraint that at least one required capability must be new at that stage. This guarantees both feasibility and genuine "new-capability pressure."

Two evaluation modes are run. In deployment evaluation, each stage starts a fresh system with no carried state, evaluated on all tasks seen so far — any degradation is purely harness-induced. In self-evolving adaptation evaluation, each system maintains persistent state across stages (memories, prompts, or learned code) and may update it using an adaptation split before being measured on a held-out split. Systems compared include ReAct (GPT-5), Codex, Claude Code, AutoGen, and DeLM, plus adaptation methods spanning memory-based (Raw Memory, ReasoningBank, MemToolAgent, G-Memory, LEGOMem), prompt-based (GEPA), and code-based (Meta-Harness) families. Metrics are pass rate, partial-credit score, cost (hours and tokens), and relative forward/backward transfer to separate new-task adaptation from retention of earlier tasks.

Why This Matters

Impact on research. The paper reframes continual learning for agents. Almost all prior work places the moving target in the task distribution or in the agent's own capability growth; EvoHarnessBench argues that in deployed systems the harness is the moving part. That shift exposes a failure mode — harness-induced forgetting — that is invisible to static-harness benchmarks and to benchmarks where the agent itself controls capability growth. It also demonstrates empirically that retention and adaptation can pull in opposite directions, which undermines the assumption that continual learning methods designed for task streams will transfer cleanly to harness evolution.

Real-world applications:

• Enterprise agent platforms (e.g., Salesforce Agentforce): as tool catalogs and specialist agents are added over quarters, agents must not silently regress on workflows they previously handled.
• Coding and software agents: as IDE/CLI harnesses gain new commands and integrations, agents need to adopt them without forgetting older workflows.
• Customer-support and CRM automation: new database tables, email templates, and escalation agents appear constantly; delegation policies learned earlier can go stale.
• Tool-augmented assistants on shared capability marketplaces: agents that select from a third-party tool registry face a catalog that changes underneath them.

Industry relevance. The paper quantifies a cost-accuracy trade-off that product teams feel directly: broader capability exposure can raise accuracy but multiplies token spend, and adaptation methods can improve retention while actively harming performance on newly introduced tasks. It also shows that memory design choices (structured vs. raw, abstract vs. concrete, schema-inclusive vs. not) have large and architecture-dependent effects, which is actionable for anyone building persistent memory into a deployed agent. The finding that task-scoped adaptation can be both cheaper and comparably effective is immediately relevant to training and adaptation budgets.

Future Directions

• Coupling outer evolution with inner adaptation. The paper explicitly motivates methods that treat harness expansion and adaptation as a joint problem rather than optimizing adaptation against a fixed harness. How to detect stale persistent state and revise it as capabilities are added remains open.
• A retention–adaptation Pareto frontier. Since positive backward transfer and positive forward transfer seem to trade off, a natural next step is characterizing and optimizing that frontier rather than reporting single-point averages.
• Better skill retrieval and engagement policies. Sparse invocation (near zero for default GPT-5) suggests the bottleneck for skills is not execution but deciding when and how to retrieve. Learned retrieval policies that scale with library size are a clear target.
• Domain-general agent delegation under expansion. The agent axis produced the largest deployment degradation and the largest adaptation gains, but with strong environment dependence. Understanding what makes a coordination environment forgiving or brittle under pool expansion is unresolved.
• Does the frequency-ranked release schedule match reality? The benchmark releases capabilities from core to long tail. Whether real-world harnesses follow that order, and whether the ordering itself changes the difficulty of the problem, is an empirical question the benchmark invites.

Target Audience

Researchers and engineers working on LLM agents, continual learning, and agent memory systems will get the most from this paper. It is also directly useful for practitioners deploying agents on platforms whose tool or agent catalogs change over time — particularly those responsible for evaluation, regression testing, or memory/retrieval infrastructure. Readers interested in multi-agent coordination will find the agent axis and its negative results on ALE especially informative. A background in benchmark design and basic continual-learning metrics (forward/backward transfer) is helpful but not required beyond the paper's own definitions.

Authors’ abstract

Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EVOHARNESSBENCH, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EVOHARNESSBENCH places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.