Can Coding Agents Sustain Continual Evolution without Forgetting?

AI summary

Overview

Research area: Software engineering (cs.SE) — autonomous coding agents and agent-harness self-evolution, with ties to continual learning.

Technical level: Intermediate. The framing is intuitive (agents forgetting old skills), but the paper assumes familiarity with coding-agent architectures (foundation model + harness), benchmark suites such as SWE-Bench, and continual-learning vocabulary like stability–plasticity and catastrophic forgetting.

Scope: An empirical study of whether agent harnesses that improve themselves on one software engineering task type can keep evolving across several task types without losing previously acquired capabilities.

What This Paper Is About

A coding agent is a foundation model plus a "harness" — the scaffolding that decides how the agent sees tasks, calls tools, and executes work. Recent work shows a harness can improve itself by analyzing its own execution traces, but those studies evolve the harness for a single task type. In real use, a developer may ask an agent to add a feature one day and write tests the next. This paper asks whether harness self-evolution survives that shift, and finds it often does not: gains on a new task type can erase most of the gains on the previous one.

Key Contributions

1. Formulates a new problem: continual harness self-evolution. The paper defines an ordered sequence of task-type datasets $D_0, \dots, D_K$ where the harness is repeatedly updated from trajectories on each dataset, and formalizes both adaptation (improve on the current task) and retention (do not degrade on earlier tasks), with absolute ($\Delta_{\text{forget}}$) and relative ($\rho_{\text{forget}}$) forgetting metrics.

2. Demonstrates catastrophic forgetting empirically across six directions. Two representative methods (AHE and Meta-Harness) are evaluated over SWE-Bench, SWT-Bench and FeatureBench, in two-stage sequences covering all six ordered directions, and nearly every direction shows substantial degradation on the earlier dataset.

3. Diagnoses the causes through manual trajectory analysis. More than 520 agent trajectories and harness code diffs are examined to produce a taxonomy of three harness-level conflict types — verification, workflow, and budget strategy — with quantified prevalence among pass-to-fail cases.

4. Isolates each conflict type in a controlled ablation. Representative modifications of each type are injected into the best SWE-Bench harness to separate their effects on accuracy from their effects on execution time.

Main Findings

• Self-evolution works well on a single task type. AHE lifts SWE-Bench from 82.0% to 96.0% over nine rounds, and Meta-Harness reaches 94.0%. On SWT-Bench both methods go from 82.0% to 100.0% on the sampled 50 tasks. On FeatureBench both achieve improvements of more than 10 percentage points.

• Continual evolution triggers severe forgetting. With DeepSeek-V4-Flash-0731, Meta-Harness reaches 94.0% on SWE-Bench after evolution, but after further evolution on FeatureBench its SWE-Bench score falls to 32.0% by the fifth round, a 66% relative decrease.

• Some directions are near-total losses. Evolving from SWE-Bench to SWT-Bench causes AHE and Meta-Harness to lose 100% and 98% of SWE-Bench performance respectively after five rounds — the resulting agents are described as almost unable to solve SWE-Bench at all. In the SWE → Feature direction, AHE and Meta-Harness drop by 29% and 66%.

• Forgetting depends on task relatedness. SWT-Bench (test generation) is the outlier; SWE-Bench and FeatureBench both require modifying source code. Evolution between SWE-Bench and FeatureBench degrades less, while evolution involving SWT-Bench degrades far more — suggesting task relatedness matters.

• Verification conflicts dominate the observed failures. Of the pass-to-fail cases, verification conflicts account for 87.1%, workflow conflicts 8.2%, and budget conflicts 0.6%, with the remainder unclassified.

• Prevalence does not equal impact. Adding a mandatory artifact constraint to the 96.0% SWE-Bench harness collapses SWE performance to 0.0 (a 100% drop) while execution time rises 69% (245.5 s to 415.7 s). A workflow conflict reduces performance to 66.0 (down 31%) and inflates execution time by 235% (to 821.5 s). A budget strategy leaves accuracy at 96.0 (0% drop) but raises execution time by 79% (to 439.7 s).

• Artifact constraints are the sharpest failure mode. Evolving SWE-Bench → SWT-Bench reverses the submission logic: the harness rejects any solution that modifies files beyond test files, which is correct for SWT-Bench but fatal for SWE-Bench, where source-code fixes are the deliverable.

• Harness choice alone shifts performance a lot. With the foundation model fixed, different harnesses range from 82.0% to 96.0% on SWE-Bench.

• Prompt paraphrasing changes what gets learned. When original SWT-Bench prompt templates are kept, Meta-Harness learns a template-dependent branch that actually helps distinguish task types; after paraphrasing, this shortcut disappears, suggesting paraphrase reduces exploitation of dataset-specific surface form.

Methodology in Plain English

The researchers built a sequence of "task types" rather than a single task. They started from Mini-SWE-Agent, a small harness (about 100 lines of Python, scoring over 74% on SWE-bench Verified), and limited what evolution was allowed to touch: the system prompt, the agent's control flow, and tools — not the model interface or the execution environment.

They picked three benchmarks covering different work: SWE-Bench Verified for fixing bugs (500 instances; 50 sampled), SWT-Bench for writing tests (50 sampled, with prompts paraphrased to break a shared sentence template), and FeatureBench for implementing features (30 sampled). Each evolution round ran the same batch of tasks twice, at temperature 0, inside an E2B sandbox with a 1-hour limit per task, using DeepSeek-V4-Flash-0731.

They then applied two existing evolution methods: AHE, a closed loop where a debugger summarizes success and failure causes and an evolve agent edits the harness code (9 rounds, 10 evaluated harnesses), and Meta-Harness, which maintains a Pareto frontier of harnesses and proposes new variants from accumulated experience (5 rounds). First they evolved on each dataset alone and recorded the best harness. Then they took that best harness, moved it to a second dataset, and kept evolving there — measuring how much the original dataset's score fell after 3 and 5 further iterations. Finally, they read code diffs and replayed failing trajectories by hand, and injected individual modifications back into the best harness to measure each one's isolated effect.

Why This Matters

Impact on research. The paper reframes harness self-evolution from a single-task optimization into a continual-learning problem, importing the stability–plasticity dilemma and catastrophic forgetting into agent engineering. It also warns that evolution can latch onto dataset-specific artifacts, such as prompt templates, rather than genuine task semantics — a methodological hazard for anyone benchmarking self-improving agents.

Real-world applications:

• Developer tooling where one agent is expected to fix bugs, add features, and write tests across a long project lifecycle.
• CI/CD and maintenance agents that accumulate harness improvements over months without regressing on older workflows.
• Multi-team platforms where a shared agent harness is retrained on one team's task mix and must not break another's.
• Benchmarks and evaluation suites that need to test retention, not just single-task peak scores.

Industry relevance. Deployment of coding agents is already broad — the paper cites the Stack Overflow Developer Survey 2026 reporting 72.9% of developers using AI agents for writing, editing, or generating code, 62.2% for debugging, and 50.6% for writing or maintaining tests. Any product that ships a self-improving harness to users with heterogeneous task mixes faces exactly the failure mode documented here.

Future Directions

• Conflict-aware evolution. The authors explicitly call for self-evolution methods that detect or anticipate verification, workflow, and budget-strategy conflicts before committing harness changes.
• Retention-aware objectives. The formal conditions in the paper require improvement on the current dataset without degradation on prior ones, but no evaluated method enforces this; how to optimize for both jointly is open.
• Understanding the task-relatedness effect. SWE-Bench and FeatureBench transfer far better than either does with SWT-Bench. What makes a task pair compatible for harness transfer, and can it be measured in advance, is unresolved.
• Robustness to dataset surface form. Paraphrasing changed what the evolver learned. How to keep evolution anchored to semantics rather than templates, and whether other shortcuts persist, remains to be studied.

Target Audience

Researchers and engineers working on autonomous coding agents, agent-harness design, and self-improving LLM systems; continual-learning researchers interested in a non-parameter-update form of forgetting; and benchmark designers or product teams who need agents to stay competent across many task types over time. Readers without background in coding-agent architectures will need to consult the cited harness work (Mini-SWE-Agent, AHE, Meta-Harness) first.

Authors’ abstract

Recently, harness self-evolution has emerged as an important paradigm for improving agent performance. Prior studies show that evolving the agent harness can effectively enhance agent capabilities without modifying the foundation model's parameters. We find that existing approaches largely study agent evolution on a single task. In practice, coding agents may need to handle multiple task types over time. Whether an agent harness can continuously evolve across multiple tasks without suffering from catastrophic forgetting remains an important open question. In this paper, we investigate this open question through a comprehensive empirical study of continual harness self-evolution. We organize three software engineering task datasets into six different evolution directions and apply two representative harness evolution methods to perform continual evolution along these directions. Our experimental results show that evolution on a subsequent dataset can substantially undermine the performance on previously seen datasets. For example, with DeepSeek-V4-Flash-0731, Meta-Harness improves its performance on SWE-Bench to 94.0% after evolution. However, after further evolution on FeatureBench, its performance on SWE-Bench drops to 32.0% by the fifth round, corresponding to a 66% relative decrease. Finally, based on our experimental findings, we identify several insights, providing directions for the continual self-evolution of agent harnesses. Code is available at https://github.com/THU-Agent/continual-harness-evolution.