Harness^2: Recursive Agent Harnessing for an Open World

AI summary

Overview

Research area: Artificial intelligence — autonomous LLM agents, test-time adaptation, and recursive self-improvement (RSI) of agent scaffolding.

Technical level: Intermediate. The paper assumes familiarity with LLM agents, tool-calling runtimes, and prompt/workflow engineering, but contains no model training or heavy mathematics.

Scope (one sentence): The paper introduces Harness², a plug-and-play framework that improves an LLM agent's execution harness for the task at hand and then learns from that improvement process to better improve harnesses on future, unseen tasks — all without ground-truth labels or weight updates.

What This Paper Is About

An LLM agent's performance depends not only on the underlying model but on its harness: the prompts, memory, skills, tools, and orchestration surrounding it. Manually engineering these harnesses for every new domain is expensive, and existing automatic approaches either tune a harness offline against fixed training tasks (which then fails to generalize) or adapt per task without carrying anything forward. Harness² tries to do both at once: it refines the harness for the current task using behavioral evidence instead of ground-truth answers, and it distills lessons from that refinement into a persistent "improvement harness" that guides how future harnesses are edited.

Key Contributions

1. Recursive agent harnessing. A formulation in which an agent both refines its current execution harness and updates the improvement harness that governs how refinement happens — extending the "Agent = Model + Harness" view from an agent that executes tasks to one that also learns how to improve its own scaffolding.

2. Task-level recursion without ground-truth feedback. A contrastive Propose–Probe–Compose step that proposes invariant and domain-specific harness edits, executes them under matched conditions, and composes a successor harness based on behavioral contrasts rather than correctness labels. It supports both parallel recursion (multiple candidates probed and composed concurrently) and sequential recursion (chained revisions).

3. Harnessing-level recursion. After each task, the framework reflects on the full improvement record and writes adopted harnesses plus supporting lessons into a harness improvement library inside the improvement harness (EXPERIENCE.md entries with an index), so later tasks start from accumulated experience rather than a tailored one-off harness.

4. An eight-component harness decomposition. Building on AHE's file-level decomposition and extending it by one component, the harness exposes eight editable components mounted as runtime files: system prompt, guardrails, memory, skills, tool guidance, scripts, subagents, and a middleware plugin.

Main Findings

• Single-step gains are consistent across all four model–harness configurations. One contrastive improvement step improves both GLM-5 and Gemini 3.5 Flash under both OpenCode and Codex on all three benchmarks, with gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB Partial pass rate.

• Weaker executors benefit more on WorkBuddy-Bench and LAB. A single step lifts GLM-5 to 75.7 under OpenCode and 75.3 under Codex on WorkBuddy-Bench, surpassing the corresponding base Gemini 3.5 Flash scores of 72.4 and 74.5. On JobBench the pattern reverses: Gemini gains 4.5 and 1.4 points versus GLM-5's 3.6 and 1.2.

• Task-level recursion adds further gains. Parallel recursion at k=5 adds a further 2.9 to 3.9 points on WorkBuddy-Bench, 3.0 to 4.2 on JobBench, and 2.6 to 6.7 percentage points on LAB Partial over a single step.

• Both recursion forms improve monotonically with budget. On WB-code, parallel recursion rises from 76.5 to 80.8 as k goes from 1 to 5; on WB-office it rises from 79.5 to 83.4. Either recursive form at k=2 outperforms every plotted baseline at k=5 on both domains. Harness Scaling regresses from k=2 to 3 on both domains, and ReasoningBank fluctuates.

• The framework sits on the compute–performance Pareto frontier. With Gemini 3.5 Flash under OpenCode, raising k from 1 to 5 increases gains over the base from 4.5 to 7.5 points on JobBench and from 5.1 to 8.4 on WorkBuddy-Bench. Meta-Harness uses more compute than Harness² at k=5 for lower performance.

• All three ablated components matter. On WorkBuddy-Bench / JobBench, removing the library costs 3.1 / 1.8 points, removing scoped editor roles costs 2.9 / 3.7, and removing composition costs 1.6 / 1.9.

• Naive refinement without the improvement harness degrades performance. Directly refining the base harness without reference executions, probes, or the library drops Code from 73.4 to 63.2 and Web from 69.9 to 65.0. A library accumulated at k=1 restores base performance; at k=5 it supports gains of 0.9, 4.4, and 5.1 points on Code, Web, and Office.

• Libraries transfer across benchmarks. A JobBench-derived library improves one-step refinement on WorkBuddy-Bench by 0.6, 0.5, and 2.9 points across its three domains. Excluding each task's own library entry still yields gains of 1.9 points on WorkBuddy-Bench and 1.7 on JobBench.

• A stronger editor is not required. Using the same model as executor and editor (OpenCode, k=2), all five models from GLM-5 to Claude Opus 5 improve on every benchmark; Opus 5 gains 3.3, 5.5, and 3.0 points on WorkBuddy-Bench, JobBench, and LAB, while GLM-5 gains 13.7 points on LAB.

• Baselines are mixed or unstable. ReasoningBank falls as much as 6.6 points below the base harness on coding tasks, and Harness Scaling underperforms the base in seven of the twelve benchmark–configuration pairs. On LAB Partial, Meta-Harness still leads for Gemini 3.5 Flash under Codex.

• Case study. On a JobBench task with Gemini 3.5 Flash under OpenCode, the base run scored 43.1; a process-level guardrail (inventory inputs, trace figures to sources, reopen outputs for verification) plus a domain skill for reconciliation mechanics raised the composed harness's score to 77.8.

Methodology in Plain English

The setup is deliberately minimal. A frozen model (Gemini 3.5 Flash or GLM-5) runs inside an off-the-shelf harness (OpenCode or Codex). The harness is a set of editable files. A separate "editor" model — Claude Opus 4.8 by default — is allowed to rewrite those files, but never touches the executor's weights.

For each new task in a stream, the framework takes three steps. Propose: it first runs the current harness once to get a reference execution, then writes two candidate harnesses — one targeting invariant weaknesses that recur across domains (planning, tool use, context management, delegation), and one targeting domain-specific knowledge, procedures, or constraints. Probe: it runs both candidates on the same task under matched settings and compares their trajectories and deliverables against the reference. This produces a behavioral contrast — evidence about whether the edit helped, did nothing, or caused a regression — not a verdict on whether the answer was correct, since no ground truth exists in the open world. Compose: the editor reads the reference, candidates, their diffs, and the evidence, then selectively merges supported changes, revises conflicts, adopts one candidate, or keeps the current harness. Each decision must be tied to a concrete observation.

This single contrastive step can be widened or deepened. Parallel recursion collects k reference executions, proposes k invariant and k domain-specific candidates, probes them all, and runs k independent composition sessions over the full pool — a width increase, not a depth increase. Sequential recursion chains k contrastive steps, each conditioned on the previous harness and accumulated history, so later steps can revise or undo earlier edits. When k>1, an agentic selector compares anonymized deliverables and adopts one; it cannot synthesize new content or see which harness produced which output.

The second level kicks in after the task's deliverable is returned. The system reviews the full improvement record — references, candidates, probes, composition decisions, and the selected result — and extracts lessons about which changes addressed which deficiencies, under what conditions, and with what consequences. Lessons are only retained when supported by execution contrasts; a good final output alone doesn't establish transferability. Adopted harnesses and their lessons enter a library inside the improvement harness. The next task still begins from the same original base harness, but refinement is now guided by that library: earlier components can be reused or adapted, and experience informs probing and composition.

Evaluation covers three professional benchmarks — WorkBuddy-Bench (office deliverables in Code, Web, and Office domains), JobBench (end-to-end professional job tasks with long executions), and Harvey's Legal Agent Benchmark (LAB, reported as per-criterion Partial pass rate) — with means and standard deviations over three runs using shuffled task orders. Baselines are the base harness, ReasoningBank, Meta-Harness, and Harness Scaling.

Why This Matters

Impact on research. The paper reframes recursive self-improvement for agents away from "search for a better artifact" toward "learn how to search." Its central empirical claim — that the process of harness refinement produces reusable knowledge, and that this knowledge can be banked and transferred — is a meaningful shift for anyone working on test-time adaptation, agent scaffolding, or self-improving systems. The result that naive refinement without that banked experience actively hurts (Code dropping from 73.4 to 63.2) is a useful cautionary finding for the field.

Real-world applications:

• Legal work: LAB is a legal benchmark; the framework could adapt document-review and drafting agents to firm-specific procedures without labeled answer keys.
• Office and back-office workflows: WorkBuddy-Bench's Code, Web, and Office domains map onto spreadsheet, memo, and small-tool deliverables common in enterprise operations.
• Professional job automation: JobBench targets long-horizon end-to-end tasks, relevant to delegating multi-step knowledge work.
• Domain-specific agent deployment: Any setting where tasks arrive unlabeled and requirements drift, such as internal support tooling or regulated document processing.

Industry relevance. The framework is plug-and-play over existing harnesses (OpenCode, Codex) and requires no model fine-tuning, which matters for organizations deploying third-party models they cannot retrain. That gains hold even when the same model acts as both executor and editor removes a dependency on a stronger, more expensive editor model. The Pareto-frontier compute result gives practitioners a knob — increase k for more gain at more cost — rather than a fixed trade-off.

Future Directions

• Longer deployment streams and library maintenance. The experiments run over benchmark task streams with libraries starting empty; how the improvement library behaves over much longer horizons — whether entries conflict, go stale, or need pruning — is not reported.
• Pushing the recursion budget further. The budget study covers k from 1 to 5, and parallel recursion is described as a single population update where k controls width rather than depth. Whether iterated population updates or larger k continue to pay off is left open.
• Cheaper and weaker editors at scale. The same-model-as-editor experiment uses OpenCode at k=2. How far this holds at larger budgets and across more model families is untested.
• Cross-domain library transfer more broadly. Transfer is demonstrated from JobBench to WorkBuddy-Bench; whether libraries transfer across more distant domains, and how much of the gain comes from reusable components versus reusable lessons, remains an open question (the paper notes component reuse is analyzed in its appendix).

Target Audience

Researchers and engineers working on LLM agents, agent harnesses, and test-time adaptation will get the most from this paper, particularly those interested in self-improving systems that cannot rely on ground-truth verifiers. It is also relevant to practitioners deploying agents in enterprise domains — legal, office, professional services — who need adaptation without model retraining. Readers seeking model-training techniques or formal RSI theory will find less here; the contribution is a practical, empirically validated framework.

Authors’ abstract

LLM agents' capabilities depend not only on their underlying models but also on their execution harnesses. Recursive self-improvement (RSI) offers a promising approach to automatically evolve these harnesses. However, deploying RSI in open-world settings still faces two critical challenges. First, open-world tasks are typically unseen during the harness design phase and novel at test time, leaving no ground-truth verifiers to reliably guide online harness edits. Second, the RSI process remains highly task-specific and fails to generalize to novel scenarios. We introduce Harness^2, a plug-and-play, recursive agent harnessing framework that addresses these challenges through two connected levels of recursion. Task-level recursion refines the current task's harness using a contrastive Propose-Probe-Compose step that reliably guides edits without ground-truth verifiers, and can be scaled further via parallel or sequential recursion. Harnessing-level recursion then extracts experience from these edits to update a continual improvement harness, enabling the agent to better adapt to future tasks. Across three professional benchmarks and four model-harness configurations, a single step consistently improves GLM-5 and Gemini 3.5 Flash over their base harnesses, yielding gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB, with further gains as the task-level recursion budget increases.