Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection

AI summary

Overview

Research area: automated optimization of LLM agent "harnesses" (the code that decides what to store, retrieve, and present to a fixed model), with a focus on reducing evaluation cost during search. Technical level: Intermediate. Scope: this paper introduces Task-CoEvolve, a method that co-evolves the validation task subset alongside the harness during optimization, matching full-set search performance while cutting evaluations by 80%.

What This Paper Is About

Automated harness optimization improves an LLM agent by rewriting the code around a fixed model, repeatedly evaluating candidate harnesses on a validation set. Existing methods evaluate the entire fixed validation set at every iteration, which is expensive and static: tasks that all candidates already solve, or that none can solve, keep consuming budget without helping to rank candidates. Task-CoEvolve attacks this by changing which tasks are used for evaluation at each iteration, using past outcomes to focus on tasks near the current capability frontier and then estimating full-set performance from the partial evaluations.

Key Contributions

1. A new problem setting. The paper frames optimizing which tasks are used to evaluate candidate harnesses as a distinct problem, orthogonal to prior efficiency work that reduces the number of candidate harnesses rather than the evaluation cost per candidate.
2. Task-CoEvolve. A method combining variance-weighted adaptive task selection (based on the Bernoulli variance of each task's historical success rate) with sampling-aware full-set score estimation (Hájek or anchored difference estimators), so candidates evaluated on different subsets can be compared on a common scale.
3. Empirical validation on two settings. Online text classification (LawBench, Symptom2Disease, USPTO-50k) and Terminal-Bench 2.1, showing Task-CoEvolve beats subset-based baselines and matches full-set search at a fraction of the evaluation budget.
4. Cost and component analysis. An ablation isolating the three components, an analysis of task discriminability across iterations, an assessment of estimator accuracy against full-validation ground truth, and measured token/time savings of 67–80% on Terminal-Bench 2.1.

Main Findings

• Task-CoEvolve achieves the highest text classification accuracy at both tested budgets. At a sampling rate of 20% it reaches 49.3% average held-out test accuracy, and at 7% it reaches 47.6%, versus 47.2% and 45.2% for the Naive fixed-subset baseline (2.1 and 2.4 points higher respectively). Both budgets use 1,560 and 480 total sample evaluations against 7,800 for full-set search.
• At 20% budget it beats full-set search. Meta-Harness with full-set search reaches 48.6% ± 0.8; Task-CoEvolve reaches 49.3% ± 0.8 using only 20% of the validation set. The authors suggest this may be because full-set search overfits the validation set during search.
• At 7% budget it nearly matches full-set search. Task-CoEvolve improves the few-shot starting accuracy from 41.6% to 47.6% while using 16 times fewer samples than full-set search.
• On Terminal-Bench 2.1, 20% budget performs close to full search. Task-CoEvolve reaches 61.8% on GPT-5.6 Luna and 41.6% on Qwen3.6-35B-A3B (51.7% average), against 62.9%, 42.7% and 52.8% for full-set search — roughly one task out of 89. It beats Naive (47.2% average) and Random-Resample (48.4% average). Starting harnesses scored 41.0% (Terminus-KIRA) and 43.8% (Terminus 2).
• Search cost drops by 67–80%. On GPT-5.6 Luna, full search used 2,888M input tokens over 22.2 hours while Task-CoEvolve used 579M tokens over 11.5 hours. On the self-hosted Qwen3.6-35B-A3B, input tokens fell from 741M to 246M and search time from 38.0 to 20.5 hours.
• Equal budgets do not mean equal cost. All 20% protocols evaluate the same number of tasks, but average input tokens per trial were 0.7M for Random-Resample, 1.2M for Naive, and 3.2M for Task-CoEvolve (comparable to full search's 3.2M), because variance-weighted selection concentrates on long-running multi-turn tasks where candidates differ. Random-Resample's 96% token reduction comes with final performance 3.3 points below Task-CoEvolve.
• Search time falls less than tokens. Time dropped 1.9 times while token usage dropped 5.0 times, because trials run with 10-way parallelism (so the slowest task dominates) and meta-agent candidate proposal takes about 2.4–3.3 hours regardless of budget.
• Ablation confirms each component contributes. Starting from Naive at 47.2% ± 0.6, adding uniform resampling gives 48.2% ± 0.5, adding full-set estimation with Ŝ-max selection gives 48.8% ± 1.4, and adding variance-weighted selection gives 49.3% ± 0.8.
• Most of the task pool is uninformative. In both benchmarks, tasks that nobody solves or that almost everybody solves consistently account for more than 70% of the pool, while the most discriminative group (historical accuracy between 1/3 and 2/3) stays a small fraction.
• Task difficulty shifts as the harness evolves. In text classification, the number of samples solved by almost all candidates rose from 34 to 58 across iterations, while on Terminal-Bench 2.1 the number of tasks no candidate could solve fell from 32 to 21.
• Estimator accuracy is budget-dependent. Re-evaluating all 60 text classification candidates on the full 130-sample validation set, the rank correlation between estimated and true scores is 0.62 at 20% budget (selected candidate ranked 12th of 60, true score 51.3% versus 54.7% for the best) and 0.13 at 7% (ranked 10th of 60, 46.4% versus 47.8%).
• Estimator choice matters and is benchmark-specific. Swapping in difference estimation on text classification lowered accuracy by 3.3 points at 20% and 3.6 points at 7%; using Hájek estimation on Terminal-Bench 2.1 produced unstable estimates (one candidate with a raw score of 33.3% was estimated at 85.9%).
• Stronger models leave less room for harness gains. Under full-set search, DeepSeek-V4-Flash started at 70.8% and the best of 10 candidates also scored 70.8% (a gain of 0.0), which the authors attribute to the benchmark and model rather than the method.

Methodology in Plain English

The search proceeds in iterations, like prior Meta-Harness work: a meta-level agent reads previous candidates and their results and writes a new harness; the harness is scored; the loop repeats. Task-CoEvolve changes only how that scoring is done.

First, before searching, two starting harnesses (zero-shot and few-shot) are evaluated on the full validation set. This is not extra cost, because the meta-agent needs them anyway; their outcomes become the initial history, giving every task a success rate.

Second, at each iteration a subset of tasks is drawn with weights. The weight is based on the Bernoulli variance of a task's past success rate — highest when a task is solved about half the time and zero when it is always solved or always failed — plus a small floor for never-solved tasks (which might become solvable) and a bonus term λ/√n_t that favors tasks with few observations so they are not dismissed prematurely. The weights are recomputed as the harness evolves, so the sampled tasks move with the capability frontier.

Third, because different iterations sample different tasks, raw subset scores are not comparable. The method estimates the full-set score using each task's inclusion probability, estimated by 4,000 Monte Carlo repetitions. Two estimators are offered: the Hájek estimator, which divides each sampled outcome by its inclusion probability and normalizes, suited to pools whose mean success rate is near 0 or 1; and an anchored difference estimator, which weights the deviation of an outcome from that task's historical success rate, suited to pools whose mean is near the middle. The paper states the choice per benchmark follows a rule based on Phase-0 evaluations and pool structure. After optimization, the candidate with the highest estimated score wins, with exact ties broken toward the earliest iteration.

Hyperparameters were held fixed across both benchmarks: weight floor ℓ = 0.125, uncertainty coefficient λ = 0.025, 4,000 Monte Carlo repetitions, and a history window of all past iterations.

Why This Matters

The work shows that the validation set itself is a controllable resource in automated agent improvement, not a fixed constant. It reduces a cost that has become a practical bottleneck: Terminal-Bench 2.1 requires tens of minutes of sandbox execution per task, so evaluation, not model inference or candidate generation, dominates the search loop. The finding that a 20% budget can beat full-set search also suggests that repeated full-set evaluation can overfit the validation set.

Real-world applications:

• Cheaper agent development pipelines. Teams building agent harnesses can iterate on the same benchmark at one-third to one-fifth of the token cost, making repeated search runs affordable.
• Long-horizon, expensive benchmarks. Evaluation regimes where each task occupies a sandbox for tens of minutes benefit most, since the method targets exactly this cost structure.
• Evaluation on small task pools. The paper notes Terminal-Bench 2.1 is small and expensive enough that a separate validation split would weaken the search signal; adaptive subsetting offers an alternative to splitting.
• Ranking candidate systems generally. The variance-weighted selection idea applies wherever many candidate configurations must be compared on a shared, costly task set.

Industry relevance: the savings are measured in input tokens (2,888M down to 579M) and wall-clock hours (22.2 down to 11.5) on a commercially relevant benchmark, with a reported USD cost drop from 22.2 to 30 for Task-CoEvolve versus full search on GPT-5.6 Luna — note the paper reports Cost (USD) values of 22.2 for full search and 30 for Task-CoEvolve in Table 3, alongside the reported "vs. Full" reductions of 80%, so the exact cost accounting should be read from the table.

Future Directions

• Dynamic evaluation budgets per candidate. The authors list as a limitation that Task-CoEvolve fixes how many tasks each candidate is evaluated on before seeing any results, so it cannot stop early on a clearly worse candidate or evaluate more tasks when two candidates are hard to distinguish. Deciding this number during evaluation is left to future work.
• Better estimators under very small budgets. At 7%, rank correlation between estimated and true scores fell to 0.13, which the authors attribute to sample size rather than the estimator; improving estimation in this regime is open.
• Combining with search-side efficiency methods. The paper positions its contribution as orthogonal to DemoEvolve, ShinkaEvolve, TurboEvolve, and HarnessCompass, which improve candidate generation and selection, leaving open how the two families of savings interact.
• Generalizing beyond the two evaluated settings. The estimator rule depends on the structure of the task pool (number of pools, mean success rate), and the paper relies on Phase-0 evaluations to guide the choice; whether this transfers to other benchmarks or to stronger models with less headroom (as seen with DeepSeek-V4-Flash scoring 70.8% before and after search) is not settled.

Target Audience

Researchers and engineers working on LLM agent systems, automated program or prompt optimization, and LLM evaluation methodology. It is most useful to readers already familiar with iterative harness/prompt search frameworks such as Meta-Harness, and to practitioners who must run expensive multi-task benchmarks and want to cut evaluation cost without losing the ability to select good candidates. Readers interested in sample-efficient evaluation (active testing, tinyBenchmarks) will also find the framing relevant, though the target of selection here is evolving harness candidates rather than a fixed model.

Authors’ abstract

We present a novel approach to efficient LLM agent harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose $\textbf{Task-CoEvolve}$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the agent's capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms fixed-subset baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.