ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost

AI summary

Overview

Research area: Automatic harness optimization (AHO) for LLM agents, specifically multi-objective optimization of task quality and token cost under noisy (stochastic) evaluation.

Technical level: Advanced. The paper assumes familiarity with multi-objective/Pareto optimization, staged evaluation pipelines, and bootstrap-based uncertainty estimation.

Scope: The paper introduces ParetoSaddler, a framework that decouples intermediate candidate promotion from final deployment selection so that an agent's prompts, tools, middleware, and control logic can be improved for both quality and cost at once.

What This Paper Is About

LLM agents are shaped as much by their "harness" — prompts, tools, middleware, control logic — as by the underlying model, and recent work automatically rewrites harnesses to improve task performance. Existing methods optimize almost exclusively for quality, yet real deployments must also control token cost, and naively applying a cost preference at every intermediate step throws away updates that look unattractive locally but enable better harnesses later. ParetoSaddler's goal is to keep such intermediate candidates alive during search, decide promotion using statistical evidence rather than raw averages, and apply the actual deployment preference only at the end.

Key Contributions

1. Problem identification: The authors identify a mismatch in staged multi-objective harness optimization between intermediate promotion and final deployment selection, where enforcing deployment preferences too early prunes updates that are useful for later optimization.

2. The ParetoSaddler framework: A staged quality–cost optimization method combining Pareto-aware promotion (a candidate is advanced if it shows credible progress in either quality or cost), repeated execution per task, and Bayesian bootstrap screening, with the deployment preference reserved for final selection.

3. Empirical validation on two benchmarks: Consistent quality-and-cost gains on ClawMark and GAIA2 using domain-disjoint training, development, and test splits, under both Pareto-set optimization and quality-constrained cost minimization, against Meta-Harness and a quality-only AutoSaddler baseline.

4. Controlled ablations: Isolating the effects of Pareto-aware promotion (versus a MinCost local gate), repeated executions (one versus three runs), and Bayesian versus mean-based screening.

Main Findings

• Joint gains on ClawMark: The quality-extreme ParetoSaddler harness improves test quality from 70.86% to 81.35% while reducing cost from $1.17 to $1.01 per task — reported as a 14.8% quality gain and a 13.7% cost reduction.

• Joint gains on GAIA2: Quality rises from 56.22% to 62.89% while cost falls from $0.78 to $0.45 per task — an 11.9% quality gain and a 42.3% cost reduction. Gains hold across all three held-out GAIA2 universes (U21 with 107 tasks, U22 with 112, U27 with 81).

• Cost optimization does not cost quality: Despite optimizing cost as well, the quality-extreme ParetoSaddler solution beats the quality-only AutoSaddler baseline by 7.49 points on ClawMark and 5.56 points on GAIA2, while also being cheaper.

• Deployment preference shifts the operating point: Under quality-constrained cost minimization, ParetoSaddler reaches $0.82 per task on ClawMark and $0.44 on GAIA2, lower than both Pareto-selected operating points, and dominates the corresponding Meta-Harness variant on both quality and cost.

• Pareto promotion preserves stepping stones: A representative ClawMark update (I-7) cuts mini-batch cost by 17.22% with a 1.33-point quality decrease, which a quality gate would reject; on development it improved both quality (72.94% vs. 72.27%) and cost ($0.88 vs. $1.13). In a controlled gate swap, Pareto promotion advanced 11 candidates versus three for a MinCost local gate, and retained an update (I-11) that raised mini-batch quality from 83.65% to 87.62% at a 10.5% cost increase, enabling later refinement.

• Execution is genuinely noisy: Across 20 ClawMark tasks and six repeated runs per task, the mean within-task coefficient of variation was 18.8% for quality and 25.6% for cost.

• Repeated runs help: Three-run optimization reaches 79.60% quality at $0.92 per task at the cost-oriented endpoint versus 76.95% at $0.95 for single-run; at the quality-oriented endpoint quality rises from 75.71% to 81.35%.

• Bayesian screening beats mean-based gates: With a mean-based gate, the cost-oriented solution reaches 76.86% at $1.09 and the quality-oriented one 76.53% at $1.15, versus 79.60% at $0.92 and 81.35% at $1.01 with Bayesian screening. Replaying the 17 ClawMark candidates, mean-based screening would promote 14 versus 12 under Bayesian screening; one rejected candidate (I-12) appeared to trade 5.55 quality points for 7.19% cost savings but was actually 2.80 quality points worse and 10.66% more expensive on development.

• Threshold stability: A sensitivity analysis places γ = 0.7 in a locally stable region on both benchmarks, supporting it as a default. Optimizer usage cost is referenced to Appendix G but not reported in the provided content.

Methodology in Plain English

The harness is kept as the only thing that changes; the underlying model's weights are frozen. Optimization runs in iterations. Each iteration picks a parent harness, runs it on a small batch of training tasks, and an LLM-based optimizer reads the execution traces and writes a modified candidate harness. The parent and candidate are then both run several times on the same screening tasks, which produces a cloud of quality and cost numbers rather than a single number each.

To decide whether the candidate deserves a fuller evaluation, the method resamples those repeated runs many times (Bayesian bootstrap, 10,000 draws), each time asking: in this resample, did the candidate improve quality without a severe cost regression, or cut cost without a severe quality loss? If at least 70% of resamples say yes, the candidate is promoted. Promoted candidates get evaluated on a held-out development set, join a pool of evaluated harnesses, and can themselves become parents later — even if they are not currently on the Pareto frontier. The actual deployment preference (a fixed trade-off, a quality floor, or a full Pareto frontier) is applied only once the optimization budget runs out.

Why This Matters

For research, the paper shows that in multi-objective agent optimization the search signal should not be the deployment objective: early rejection of locally unattractive updates can trap the search in a cost local minimum. It also shows that ignoring execution noise in promotion decisions measurably degrades the final harness, giving a concrete recipe (repeated rollouts plus bootstrap support) that other AHO pipelines can adopt.

Real-world applications:

• Coding and developer agents that run many long tool-using sessions, where per-task cost directly determines whether a product is economically viable.
• Customer-support and back-office automation, where a quality floor must be met under a per-ticket cost budget.
• Enterprise research and search agents that chain many model calls, where premature termination looks cheap but silently destroys quality.
• Agent platform providers who need to offer a menu of quality–cost operating points rather than one fixed configuration.

For industry, the framing maps directly onto deployment decisions: teams typically want a Pareto frontier of harness variants plus a policy for choosing among them, and ParetoSaddler produces exactly that while treating the optimizer's own token spend as a fixed, reported cost.

Future Directions

• Broadening the objective set: The framework handles two objectives; extending it to latency, safety, or privacy constraints (explicitly listed as out of scope here) is an open problem.
• Online versus offline adaptation: The method operates on a fixed development corpus; adapting harnesses during live deployment with drifting task distributions is untested.
• Hyperparameter transfer: γ = 0.7 was stable on these benchmarks, but the authors suggest a small grid search may be needed in other domains — how to set promotion thresholds and the local allowances (ε_Q, ε_C) without extra compute is unresolved.
• Generalization beyond the tested instantiations: Results cover ClawMark and GAIA2 with two specific harnesses and models; whether the promotion statistics hold for other AHO backends and much longer horizons remains to be shown.

Target Audience

Researchers and engineers working on LLM agent systems, automatic prompt/harness optimization, or multi-objective optimization under noisy evaluation. It is most useful to practitioners who already run agent evaluation pipelines and are deciding how to trade off task quality against inference cost, and to readers interested in how statistical screening can be folded into iterative search loops.

Authors’ abstract

Harness optimization can substantially improve LLM agents without modifying model weights, but existing methods primarily optimize response quality. Extending them to joint quality--cost optimization introduces a challenge: applying the deployment preference during intermediate updates can prematurely reject candidates that are valuable for later improvements. We introduce ParetoSaddler, which addresses this mismatch by separating intermediate candidate promotion from final deployment selection. At each iteration, an optimizer updates a parent harness to produce a candidate, and then compares the two over repeated executions and uses Bayesian bootstrap Pareto support to promote candidates showing credible progress in either quality or cost. Promoted candidates receive broader development evaluation and remain available as parents for later optimization steps. We evaluate ParetoSaddler on ClawMark and GAIA2 using domain-disjoint splits. On ClawMark, ParetoSaddler improves quality from 70.86\% to 81.35\% while reducing cost from \$1.17 to \$1.01 per task; on GAIA2, it improves quality from 56.22\% to 62.89\% while reducing cost from \$0.78 to \$0.45 per task. The Pareto-selected harnesses also achieve higher quality at lower cost than quality-only harness optimization. Analysis shows that Pareto-aware optimization retains intermediate candidates that would be discarded by deployment preference but can support later improvements, while Bayesian bootstrap screening filters out candidates whose gains are weakly supported across runs.