SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness

AI summary

Overview

• Research area: LLM agent systems — specifically automated optimization of the agent harness (the layer that mediates between a model and its environment: tool schemas, context management, observation handling, delegation).
• Technical level: Intermediate. The paper assumes familiarity with LLM coding agents, tool calls, prompt caching, context windows, and token pricing, but its core argument is conceptual rather than mathematically dense.
• Scope: The paper describes an RSI-inspired ("recursive self-improvement") auto-research pipeline that searches across many executable environments to discover token-efficiency mechanisms for an agent harness, then evaluates the resulting harness (SoL-Pi) on held-out benchmarks against Codex, Claude Code, Pi, and several third-party harnesses.

What This Paper Is About

Long-running coding agents generate long trajectories of reasoning, tool calls, and feedback, so token cost per task becomes a first-order systems problem. The authors argue that improving the harness — not the model or the serving infrastructure — is an orthogonal way to cut that cost without any additional model training. Their goal is to automate harness discovery: have an AI research agent inspect execution traces, propose harness changes, and retain only those that preserve task capability while improving token efficiency, with final validation kept strictly separate from the search.

Key Contributions

1. A broad-to-deep auto-research pipeline for harness discovery. The search runs roughly 150 proposed directions and roughly 500 executable environments, comprising more than 3,000 runs and more than 60,000 agent–environment interactions. It scales through isolated, disposable search lineages with three stated principles: breadth and depth, independent validation, and scalable orchestration.

2. A strict isolation boundary between search and evaluation. Capability metrics, tolerances, and efficiency metrics are fixed before experimentation and kept outside the optimizing agent's control. Held-out results never feed back into the search loop; a failed validation rejects a candidate without triggering further optimization. Of EdgeBench's 51 public tasks, 11 are used for one-way acceptance of frozen candidates and the remaining 40 are reserved for final evaluation.

3. Four retained, reusable mechanisms. Action Fusion (action execution), Online Context Compact (context compaction), ObservationPack (observation handling), and Evidence-Preserving Reducer (delegated reading).

4. Empirical evidence of cross-backend transfer. A harness developed entirely on GPT-5.6 Sol trajectories was applied to Opus 5 with no further search or adaptation, retaining 94.3% of Pi's average score while reducing token traffic by 44.7% and API cost by 33.5% relative to Pi's point estimates.

Main Findings

• EdgeBench efficiency result: On the 51-task EdgeBench evaluation, SoL-Pi achieves performance comparable to Pi across GPT-5.6 Sol and Opus 5 while reducing recorded token traffic by 44.7–49.0% and API cost by about one third. The authors estimate hourly savings of $8.75–$13.50 relative to native Codex and Claude Code harnesses, and $4.36–$5.71 relative to Pi. Figure 1(b) reports a 50.0% API cost reduction relative to Codex on GPT-5.6 Sol and 54.3% relative to Claude Code on Opus 5.

• Two reported operating points under GPT-5.6 Sol: SoL-Pi [Efficiency] uses 1.0990 B total tokens (49.0% fewer than Pi) and costs $894 while retaining 93.7% of Pi's average score (42.003 vs. 44.833); its token cost is 33.2% lower than Pi's. SoL-Pi [Performance] — the single-mechanism configuration with the highest average score, which is ObservationPack under GPT-5.6 Sol — raises average score from 44.833 to 47.208 (a 5.3% gain) while reducing token traffic by 6.1% and improving token efficiency by 9.8%.

• Comparison table context (Table 1, GPT-5.6 Sol): Codex records 3.0537 B total tokens, $1,787, average score 34.738, token efficiency 1.0086; OpenSquilla 1.3353 B, $1,243, 24.506, 0.9945; Oh-My-Pi 2.2235 B, $1,832, 26.921, 1.3347; OpenCode 2.5668 B, $3,422, 29.552, 2.2704; Oh-My-Opencode 2.5825 B, $2,678, 38.523, 1.3633; Pi 2.1538 B, $1,339, 44.833, 0.5855. The EdgeBench official GPT-5.5 @2h checkpoint (31.2) is reported as an unranked score-only reference.

• Held-out backend (Opus 5, Table 2): Claude Code records 2.0045 B total tokens, $2,535, average score 43.689, token efficiency 1.1377; Pi 2.3697 B, $1,741, 44.756, 0.7625; SoL-Pi [Efficiency] 1.3101 B, $1,158, 42.224, 0.5376; SoL-Pi [Performance] (Action Fusion for Opus 5) 2.1016 B, $1,605, 50.482, 0.6235.

• Terminal-Bench 4: On 63 CPU-only tasks, Codex and Pi each solve 18 tasks while SoL-Pi solves 15. SoL-Pi reduces total model cost by 26.3% ($211.12 vs. $286.45 for Pi) and cost per solved task by 11.6% ($14.07 vs. $15.91). Codex's totals are $272.35 and $15.13 per solved task.

• IMO 2026 (GPT-5.6 Sol at xhigh, Lean 4 verification): SoL-Pi passes 3 of 6 problems at a total model cost of $62.69, the lowest cost per passed problem at $20.90. Codex passes 5 of 6 at $114.47 total and $22.89 per passed problem; Pi passes 3 of 6 at $75.95 total and $25.32 per passed problem. Each problem is capped at 150 minutes.

• Agent swarms: In a kernel-optimization experiment measured in simulated machine cycles, starting from a frozen starter requiring 147,734 cycles, the SoL-Pi swarm (Codex coordinator plus 20 SoL-Pi workers using all four mechanisms) reaches 1,127 cycles at $60.11, compared with 1,333 cycles at $39.20 for a single Codex agent and 1,366 cycles at $82.12 for the Pi baseline swarm. The SoL-Pi swarm reduces API cost by 26.8% relative to the Pi baseline swarm, while the single agent remains least expensive. The SoL-Pi swarm and the single agent pass all eight speed thresholds; the Pi baseline swarm passes seven, missing the final threshold of fewer than 1,363 cycles.

• Every component helps in isolation (Table 4): Under GPT-5.6 Sol, adding any single mechanism to Pi's baseline (2.1538 B total tokens, $1,339, 44.833, 0.5855) reduces total tokens: Action Fusion (1.8968 B, $1,235, 46.664, 0.5190), Online Context Compact (1.2881 B, $935, 41.993, 0.4365), Evidence-Preserving Reducer (1.9375 B, $1,200, 44.630, 0.5274), and ObservationPack (2.0224 B, $1,271, 47.208, 0.5280). The same holds under Opus 5, where Action Fusion scores highest at 50.482. The complete stack has the lowest total token count and token cost in both backend blocks.

• Cache trade-off is real and measured: Shortening context can sacrifice prompt-cache reuse, and cached input also costs money. Under GPT-5.6 Sol, the complete stack reduces cache-read traffic from 2.1326 B to 1.0605 B tokens while cache-write traffic rises from 0.0141 B to 0.0316 B. Despite the extra cache writes, total model cost falls from $1,339 to $894. The authors therefore argue for evaluating full task cost rather than cache reuse alone.

• Activation differs by backend: Mechanism trigger rate (fraction of tasks on which a mechanism activates) and trigger intensity (mean activations per triggered task) are both lower under Opus 5, which the authors suggest may reflect the harness being optimized exclusively on GPT-5.6 Sol trajectories. Every configuration under Opus 5 still improves token efficiency on its triggered tasks.

• Combining mechanisms shows a complementarity pattern: ObservationPack becomes more selective in the full stack, potentially due to overlap with Evidence-Preserving Reducer on observation-heavy trajectories. For every mechanism, the full stack achieves a larger token-efficiency gain than the standalone configuration on their respective triggered-task subsets. The authors state that this does not isolate interaction effects.

• Action Fusion lineage as a case study: Its recorded lineage spans 27 iterations across four stages (oracle analysis, baseline construction, prompt and tool-schema optimization, final held-out validation). Oracle analysis projected an 11.5% token reduction under full triggering. Because prompt-only triggering proved unreliable, the agent extended the tool schema to expose the fused action directly, and introduced trigger rate alongside task score as an intermediate acceptance metric — a mechanism-specific metric the agent developed on its own.

• Conclusion-level summary of gains: The best-performing candidates improve model performance by 5.3–12.8% and token efficiency by 9.8–18.2%, while the complete stack reduces token traffic by 44.7–49.0% and token cost by about one third at comparable performance.

Methodology in Plain English

The authors treat harness improvement as a search problem and hand the search to an AI research agent. A separate agent runs the base harness in prepared development environments; the research agent then inspects those execution traces, identifies recurring sources of wasted work, and proposes candidate harness changes.

Each candidate is developed in its own isolated, disposable lineage. The pipeline starts from 152 proposed directions organized into six proposal families — context, progress, tools, delegation, prompt and policy, and improvement and evaluation — and these families classify where a hypothesis came from rather than constraining where the fix is implemented. An "Oracle Analysis" step examines existing trajectories to find avoidable work before any rollout budget is assigned. Each lineage follows a conventional autoresearch cycle (propose, implement, run a fixed experiment, inspect, retain/revise/discard), extended with an iterative implementation loop based on the Ralph Loop: an implementer refines the candidate to an explicit completion criterion, and an independent reviewer checks it before evaluation. Multiple analyzers each examine one trajectory for repeated actions, context growth, large observations, or sparse diagnostic signals, and a reducer merges their findings into a candidate-level summary.

Acceptance is gated. Capability metrics and tolerances are fixed up front and kept out of the optimizing agent's reach. A candidate must (1) keep every capability metric within its predeclared tolerance and (2) improve at least one declared efficiency metric; among survivors, nondominated results are retained.

Development used 535 executable environments: 495 repository tasks derived from GitHub issue–pull request pairs (with the accepted patch and regression tests hidden, and only environments whose tests fail before the patch and pass after retained) plus 40 synthetic tasks with executable Terminal-Bench-2-style verifiers. Four mechanisms survived selection, were integrated into a single harness, and had their hyperparameters and implementation refined while preserving capability. Before any held-out evaluation, the mechanism source, configuration, metrics, capability tolerances, and acceptance rule were frozen. All four mechanisms are implemented as extensions to Pi, and Evidence-Preserving Reducer uses GPT-5.6 Luna at high to extract evidence.

Why This Matters

Impact on research. The paper tests a worry that has been raised about automated harness search: that evolved harnesses overfit the tasks used during search and show only marginal gains on unseen tasks. SoL-Pi's contribution is a structural answer — freeze the acceptance rule, isolate held-out evidence, and only then evaluate — plus preliminary cross-backend transfer evidence. It also proposes viewing harness optimization as analogous to pretraining, where a harness is "exposed to many tasks and updated from the resulting trajectories."

Real-world applications:

• Coding agents in production. Unattended, around-the-clock agents that run long tool-use trajectories could be run at roughly one third lower API cost at comparable capability on the reported EdgeBench setting.
• Multi-agent swarms. In the reported kernel-optimization run, a fixed two-hour budget with 20 SoL-Pi workers produced a better result (1,127 cycles) than 20 baseline workers (1,366 cycles) at 26.8% lower API cost than the Pi baseline swarm.
• Cost-sensitive evaluation harnesses. The Terminal-Bench 4 and IMO 2026 results show the cost-per-solved-task and cost-per-passed-problem framing that budget-constrained teams can reuse, even where raw solve counts are lower.
• Formal-math agent pipelines. The IMO 2026 evaluation required Lean 4 formalization and verification, showing the harness is evaluated in a setting with machine-checked correctness rather than only subjective scoring.

Industry relevance. All three author affiliations are NVIDIA, NTU, and MIT. The claim that harness-level optimization improves efficiency without additional model training is directly actionable for anyone paying per token for long-horizon agents, and it complements — rather than competes with — faster kernels, quantization, and cheaper-model routing. The paper also notes that all token costs use API prices as of August 17, 2026, and that API prices change over time.

Future Directions

• "Pretraining the harness." The authors hypothesize that scaling both the number of executable environments and the diversity of research ideas can yield sustained gains in generalizing harness artifacts discovered through RSI, and state they will continue exploring this.

• Multi-backend training. The harness was updated from trajectories generated by a single LLM, and its mechanisms trigger less often and less intensively on Opus 5. Training and validating the harness across multiple backends may improve robustness while preserving task quality and efficiency.

• Recursive efficient improvement. The authors plan to use SoL-Pi as the starting harness for the next research cycle, on the idea that lower per-run costs could let a fixed budget cover more executable environments, trajectories, and research ideas. They explicitly frame this as a long-term vision, not a compounding effect demonstrated by the present study.

• Search coverage and cost. Running complete auto-research loops is computationally expensive, making controlled comparisons of search breadth and depth under a fixed budget difficult. The authors call scaling laws along these dimensions a promising direction and leave systematic investigation to future work.

Target Audience

This paper is most useful to engineers and researchers building or operating LLM agent harnesses — people who control tool schemas, context compaction, observation handling, and delegation, and who pay per token for long-horizon agent runs. It also speaks to groups working on automated agent design and self-improving systems, to evaluation researchers interested in separating search feedback from held-out validation, and to infrastructure and platform teams who need cost-per-solved-task figures (Terminal-Bench 4, IMO 2026) rather than capability scores alone. Readers without background in agent tool use and prompt caching will need to consult the cited prior harness work first.

Authors’ abstract

As coding agents move from supervised code completion to unattended, around-the-clock exploration, their work expands from isolated predictions into long trajectories of reasoning, tool use, and feedback. Token efficiency therefore becomes important for scaling recursive self-improvement. We take an RSI-inspired approach at the harness layer, scaling auto-research loops across increasingly numerous and diverse environments for harness rollouts. At this scale, the process yields reusable improvements that transfer beyond their development setting, moving automated harness discovery toward production-level outcomes. Four mechanisms survive selection and form SoL-Pi, spanning action execution, context compaction, observation handling, and delegated reading. On the 51-task EdgeBench evaluation, SoL-Pi achieves performance comparable to Pi across GPT-5.6 Sol and Opus 5 while reducing recorded token traffic by 44.7-49.0% and API cost by about one third. In other words, estimated hourly savings are \$8.75-\$13.50 relative to native Codex and Claude Code harnesses, and \$4.36-\$5.71 relative to Pi.