StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments

AI summary

Overview

Research area: LLM agent engineering for enterprise environments — specifically, automated optimization of the "harness" (the executable scaffolding of prompts, tools, skills, and control logic) surrounding a frozen language model.

Technical level: Advanced. The paper assumes familiarity with tool-calling agents, MCP (Model Context Protocol) providers, subagent architectures, agent loops, and benchmark-based evaluation of stateful workflows.

Scope: The paper introduces StarHarness, a framework that searches over environment-specific harness configurations while keeping model weights fixed, and evaluates it on three stateful enterprise benchmarks (ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance) for full-benchmark gains, held-out generalization, and cross-model transfer.

What This Paper Is About

Enterprise agents must operate through stateful backends with large tool surfaces, cross-step dependencies, and domain conventions that tool schemas often omit — so a capable model can still fail because the harness around it does not match the environment. StarHarness addresses this model–environment mismatch by evolving the harness (prompt and task framing, tool interfaces, skills, MCP-backed providers, subagent structure, and agent-loop configuration) without touching the model's weights. The goal is to show that a compact, carefully stratified search over harness edits produces large, generalizable, and transferable performance gains in tool-rich enterprise tasks.

Key Contributions

1. Efficient stratified harness evolution. A protocol that searches over compact task subsets stratified by baseline failure mode, separating proposer-visible search tasks, proposer-hidden selection tasks, and held-out evaluation — providing a direct measure of generalization rather than conflating search with generalization.

2. Task and model transfer of specialized harnesses. Across three stateful enterprise benchmarks, harnesses evolved with one model improve tasks excluded from evolution and transfer without re-evolution across both GPT and Qwen model families.

3. Analysis of learned specialization. Identification of three recurring forms of environment specialization — interface repair, environment conventions, and operational knowledge that compresses search — linked to measurable changes in agent precision, convergence, and efficiency.

Main Findings

• Large full-benchmark gains from a fixed model. Harness evolution improves full-benchmark performance by 20–35 percentage points over the default harness, after 4–12 accepted changes per environment.

• Gains exceed prompt optimization alone. Relative to GEPA prompt optimization applied on top of the Pi harness, StarHarness gains are +13.8, +22.3, and +17.6 percentage points on ITBench, EnterpriseOps-Gym, and AutomationBench respectively. The paper describes this comparison as descriptive, since the systems differ in prompts, tools, execution policies, and harness architecture.

• Held-out generalization is positive on all three benchmarks. For GPT-5.4, absolute gains in percentage points on the evolution set versus held-out tasks were: ITBench +45.0 vs +31.7; EnterpriseOps-Gym +22.0 vs +15.1; AutomationBench +23.0 vs +29.3.

• Frozen cross-model transfer works. Using a single StarHarness artifact evolved with GPT-5.4 and no re-evolution, every transferred model improves. Examples: ITBench Qwen3.5-27B 25.6% → 70.0% (+44.4 pp); ITBench GPT-5.4-mini (medium) 33.1% → 79.4% (+46.3 pp); EnterpriseOps-Gym Qwen3.6-27B 18.2% → 38.8% (+20.6 pp); EnterpriseOps-Gym GPT-5.5 (high) 37.8% → 48.5% (+10.7 pp); AutomationBench Qwen3.6-27B 48.2% → 75.5% (+27.3 pp); AutomationBench GPT-5.5 (medium) 59.6% → 84.9% (+25.3 pp).

• Cost per task falls alongside accuracy. Estimated GPT-5.4 inference cost per task decreased by 17% on ITBench, 53% on EnterpriseOps-Gym, and 29% on AutomationBench at published rates.

• ITBench traces show better precision, not just more work. Task score rose from 40.0% to 75.0%; turns per task fell from 25.2 to 22.1; shell calls fell from 49.8 to 46.7; false positives fell from 0.79 to 0.33; true positives rose from 0.45 to 0.78. The paper notes regressions occurred when upstream search continued after a proximate cause should have ended it.

• EnterpriseOps-Gym shows shorter workflows. Task success rose from 23.3% to 43.7%; turns per task fell from 18.12 to 9.87; tool calls fell from 29.53 to 16.83; verifier pass rate rose from 34.5% to 72.8%; cost per task fell from $1.23 to $0.58.

• AutomationBench shows fewer unsafe executions. Domain objective score rose from 57.1% to 83.2%; mean partial credit rose from 67.3% to 86.1%; turns per task fell from 16.35 to 11.98; tasks with guardrail violations fell from 20 to 4; total guardrail violations fell from 33 to 4; zero-score tasks fell from 24 to 6.

• Search mode matters structurally. Of 21 accepted patches across the three runs (4 ITBench, 12 EnterpriseOps-Gym, 5 AutomationBench), 8 on EnterpriseOps-Gym came from the tree-search exploration stage and 4 from the subsequent hill-climbing stage. The paper notes these stages are sequential, so the design describes complementarity rather than a causal head-to-head comparison.

• External context points. On EnterpriseOps-Gym, Codex scores 41.7% versus 43.7% for StarHarness, and external Claude reference scores are reported as 48.1% (Fable 5), 35.9% (Sonnet 5), and 35.5% (Opus 4.8 max). The Claude runs are external references not part of the controlled comparison; the Finance-100 subset and Stirrup harness differ from the AutomationBench paper's default setup, so those scores are not directly comparable to the benchmark paper's reported numbers.

Methodology in Plain English

The researchers treat the harness — everything around the model, such as prompts, tool definitions, argument preprocessing, skills, MCP providers, subagent structure, context management, verification, and finish logic — as the thing to optimize, while the model weights and the benchmark stay fixed.

An optimizer agent (a coding harness built on Oh My Pi, a variant of the Pi agent harness) proposes changes to a separate agent harness called Stirrup. Before evolution, the benchmark is split three ways. A compact evolution pool is sampled, typically about half the reproducible tasks, and described by three baseline measurements: failure mode (for example wrong_tool, context_loss, missing_evidence, premature_conclusion), baseline task score, and verifier pass rate. That pool is split into search tasks the proposer can see traces for and selection tasks the proposer cannot see at all, matched on those baseline distributions. The remaining tasks form a holdout set that never influences proposals or acceptance decisions.

Each iteration runs three stages. The proposer reads the current harness, the search-set traces, and a persistent memory ledger, then returns one bounded candidate patch. A validator checks scope, imports, and a single-task smoke test. A proposer-selected test-flip gate then checks whether the candidate flips one specific task; if not, it is rejected cheaply without running the full selection evaluation. Passing candidates are scored on the hidden selection set and only committed if the selection mean improves, with verifier rate used as a tie-breaker when available. Rejected, invalid, or crashed candidates revert to the prior frontier.

Two search procedures share the same proposer, validator, evaluator, and acceptance rule. Hill climbing keeps a single frontier harness and applies bounded local edits. Tree search keeps a set of candidate nodes, each with a parent pointer, cumulative patch, traces, validation status, and score, allowing the proposer to explore, draft, debug, merge, or improve alternatives before the best node becomes the frontier. Guardrails prohibit branching on task IDs or hard-coded answers, verifier or assertion content in agent prompts, ground-truth tables or hidden-state access, and benchmark-specific answer mappings, so edits must target reusable environment behavior.

Evaluation uses ITBench SRE (40 Kubernetes root-cause analysis scenarios from the OpenTelemetry demo application, with an offline snapshot of alerts, events, traces, metrics, and topology), EnterpriseOps-Gym ITSM (103 ITSM oracle tasks across incident, problem, change, knowledge, and user management, backed by a ServiceNow MCP server and graded by SQL verifiers on final database state), and AutomationBench Finance (100 finance workflow tasks across 47 simulated SaaS applications, graded by programmatic assertions on environment state). Harnesses were evolved with GPT-5.4 at medium reasoning as both agent and proposer, then evaluated frozen on additional GPT and Qwen models including Qwen3.6. Baselines include the unmodified Stirrup framework, standalone Pi and Codex harnesses, and GEPA prompt optimization on top of Pi.

Why This Matters

Impact on research. The paper pushes harness-level optimization toward stricter evaluation, arguing that search performance and generalization should be measured separately. It contrasts its task-level separation, proposer-hidden selection, and held-out evaluation with Meta-Harness, which searches and reports final TerminalBench-2 performance on the same 89-task benchmark (framed by its authors as a discovery setting). It also positions harness evolution as an external, testable, revertible code change that sits alongside — rather than replaces — model scaling.

Real-world applications.

• IT operations and site reliability engineering: diagnosing root causes across alerts, events, traces, metrics, and topology in Kubernetes environments.
• IT service management: incident, problem, change, knowledge, and user-management workflows executed against a live ServiceNow-style backend with database-level verification.
• Finance and accounting operations: accounts payable/receivable, expenses, reporting, and bookkeeping workflows spanning dozens of simulated SaaS applications.
• Enterprise agent deployment in general: lowering inference cost per task while raising task success, which matters when agents run at volume.

Industry relevance. Enterprises deploying agents against stateful backends face persistent model–environment mismatch that prompt engineering alone does not fix. The paper shows that environment-specific harness adaptation can add substantial performance beyond prompt optimization, transfer across model families without re-evolution, and reduce cost per task — all properties that matter for vendor-neutral, multi-model enterprise deployments.

Future Directions

1. Co-evolving harness and model weights. The authors propose optimizing the scaffold and the policy jointly through reinforcement learning so both specialize to enterprise interaction protocols, potentially yielding smaller, more efficient enterprise models.

2. Testing whether joint harness–weight optimization can match or exceed larger models at lower inference cost.

3. Isolating the causal contribution of individual patches. The paper repeatedly states that the paired comparisons cannot isolate the causal effect of any single accepted edit or any individual tool, leaving patch-level attribution as an open question.

4. Reducing residual failure modes. Regressions on ITBench occurred when upstream search continued after a proximate cause should have ended it, suggesting termination and convergence control as a further target for harness evolution.

Target Audience

This paper is most useful to agent-engineering researchers and practitioners who build or operate tool-calling LLM agents against stateful enterprise systems: platform and infrastructure engineers, IT service management and finance automation teams, and evaluation researchers interested in measuring generalization separately from search. It also suits readers studying prompt optimization, agent architecture search, and harness-level adaptation who want a concrete methodology and a strict evaluation protocol, though the MCP, harness, and enterprise-benchmark concepts make it advanced rather than introductory reading.

Authors’ abstract

We present StarHarness, a framework for evolving environment-specific agent harnesses while keeping model weights fixed. The evolved harness can include prompt and task framing, tool interfaces, skills, MCP-backed providers, subagent structure, and agent-loop configuration. StarHarness constructs a compact evolution pool by stratifying tasks according to baseline failure behavior, separates proposer-visible search tasks from proposer-hidden selection tasks, and reserves held-out tasks for evaluating generalization. Across ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance, harness evolution improves full-benchmark performance by 20-35 percentage points over the default harness after 4-12 accepted changes per environment. These gains persist on tasks excluded from evolution and transfer without re-evolution across GPT and Qwen model families. Trace analysis links the improvements to interface repairs, environment conventions, and operational knowledge that compresses search, with fewer false-positive diagnoses and shorter trajectories in several settings. StarHarness therefore offers a practical way to reduce persistent model-environment mismatch in tool-rich enterprise tasks.