DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneck

AI summary

Overview

Research area: LLM-driven autonomous discovery and agentic search for code and system optimization, spanning software repositories, GPU/TPU kernels, and CPU chip design. Technical level: Intermediate (readable with basic familiarity with benchmarks and speedup metrics; the systems and architecture case studies assume some background). Scope: The paper proposes DuoDiscover, a bottom-up fleet of coding agents that co-discover what region of code to optimize and a cheap, validated proxy evaluator for it, so that search can scale when the true evaluation is expensive.

What This Paper Is About

In real engineering settings, a single true evaluation of progress (a full benchmark, a serving trace, a simulator run) can take minutes to hours, so a search can only afford a handful of true evaluations. Prior autonomous discovery work largely treats the evaluator as fixed and focuses on improving the search harness instead. DuoDiscover's goal is to jointly discover what to optimize and how to measure progress cheaply, so that autonomous optimization scales to repository-level software, GPU/TPU kernels, and chip architecture design.

Key Contributions

1. Co-discovery of targets and yardsticks. Each worker agent picks its own target (a region of code, such as a function or module) and writes its own cheap proxy evaluator, called a yardstick, then hill-climbs the yardstick rather than the true objective. The agent also records a rough guess $w_T \in [0,1]$ of the share of the true score the target accounts for, bounding achievable gain by Amdahl's law to $1/(1-w_T)$.
2. Fidelity testing against reward hacking. Because search can exploit the gap between a yardstick and the true objective, the orchestrator re-tests every yardstick after every true evaluation using Kendall's τ rank correlation on pairs of states whose true scores differ by more than the measured evaluation noise, with a threshold of τ ≥ 0.5. Drifting yardsticks are invalidated and the agent either rebuilds it or moves to a new target.
3. A bottom-up fleet with a lightweight orchestrator. There is no central decomposition of the problem. An LLM orchestrator only manages the expensive true evaluations: it terminates stalled agents, spawns new ones (fresh, seeded with a transfer of the best agent's confirmed state, or as a combine that re-implements several agents' confirmed changes), and briefs running agents on what the rest of the fleet found.
4. Demonstration across the computing stack. Results on two repository-level benchmarks (FormulaCode, SWE-fficiency), four kernel suites (SOL-Exec, FA4, JAXBench, TPU-Inference), and five real-world case studies including vLLM, FreeToken, SQLite, zstd, and a DPC4 CPU prefetcher design task.

Main Findings

• Benchmark sweep: Under an 8-hour, four-agent budget, DuoDiscover scores above every baseline on every benchmark in Table 1. FormulaCode 2.156× (vs Claude Code 1.778×, PerfAgent 1.774×, OpenEvolve 1.161×); SWE-fficiency 15.02× (vs 12.81×, 12.27×, 3.76×); SOL-Exec 79.74% SOL (vs KDA 70.22%, Claude Code 67.31%, PerfAgent 64.10%, OpenEvolve 52.36%); FA4 11.97% (vs KDA 8.93%, AVO 4.11%, cuDNN attention 1.30%); JAXBench 4.62× over XLA (vs PerfAgent 4.08×); TPU-Inference 2.68× over tuned Pallas kernels (vs Claude Code 2.38×).
• Human-expert comparison: On SWE-fficiency, DuoDiscover's 15.02× geometric-mean speedup exceeds the expert patches at 6.90×. On FormulaCode it reaches 2.16× against human expert PRs at a 1.048× geometric mean, a 21% gain over the strongest agentic baseline, while modifying on average 22 files, 51 functions, and 1,800 lines per task (PerfAgent: 14 files, 800 lines; OpenEvolve: 2 files, 250 lines).
• Leaderboard standing: On the 235-kernel SOL-Exec benchmark (leaderboard state September 10, 2026), DuoDiscover ranks first on 121 kernels, second on 61, and third on 30, at a mean 79.74% SOL, ahead of several proprietary entries.
• Yardsticks are much cheaper and mostly faithful: Median yardsticks run in 40 s versus 15 min for true evaluation on SWE-fficiency (22× faster) and 16 s versus 14 min on SOL-Exec (53× faster). Validation coverage is 5.3 of 9.2 repository targets and 11 of 15 kernel targets per task, with median Kendall τ of 0.66 and 0.76; 41% of repository and 45% of kernel yardsticks are later invalidated for drift, which wastes budget but never counts a false gain.
• Search efficiency: On kernels, DuoDiscover matches Claude Code's final 69.12% score in 2.8 hours using 0.93M tokens instead of 3.6M (74% fewer) and 13 true evaluations instead of 76 (83% fewer), cutting run cost from $309 to $79.60, and ends at 80.80% (4.4M tokens, 54 true evaluations), leading on 41 of 50 kernels. On repositories it passes Claude Code's 8-hour score in a median of 2.5 hours.
• Mechanism of the gain on repositories: On the 20-task sample, Claude Code crowds onto 2 to 4 targets per task and true-evaluates roughly 72 candidates to land 6 confirmed gains; DuoDiscover isolates about 9 targets, builds a yardstick for each, tests 358 candidates on yardsticks, sends only 59 to true evaluation, and roughly doubles confirmed gains to about 12 per task.
• Real-world systems: On vLLM (GLM-5.2 in NVFP4 on an 8×B200 node, four SLOs, 85-minute true evaluation), Claude Code ends at 1.07× after 12 true evaluations while DuoDiscover reaches 1.92× after 7, with 2.94× lower p99 decode latency, 3.61× lower first-token latency, 21% lower power, and unchanged throughput from a roughly 500-line system-level change. On FreeToken (753B-parameter MoE, 433 GB of weights, single 48 GB L40S), DuoDiscover reaches 1.17× at batch 1 and 1.76× at batch 8 (1.44× geometric mean) after 6 true evaluations versus Claude Code's 1.05× after 14. On SQLite, 1.38× versus 1.12× with 17 versus 45 true evaluations; on zstd, a 1.16× instruction ratio versus 1.05× with 7 versus 21 true evaluations.
• Chip design: On DPC4 prefetcher design, where one true evaluation takes up to 28 hours over 1,700 simulations, DuoDiscover delivers an overall +6.31% gain over the reference core, versus +3.82% for AlphaEvolve-based ArchAgent v2 and +3.52% for the best human design (BertiGO), using about 8,000 candidates over 16.7 days against over 12,000 candidates and more than 40 days for ArchAgent v2. On the Multi-Core configuration it reaches +3.93% versus +2.85% for the best human design and +1.33% for ArchAgent v2, which the authors describe as the first autonomous system to beat the human best on those multi-core cases.
• Variance and ablations: Across three seeds, final scores stay within ±1.12% of the per-kernel mean and ±1.8% of the per-task mean (Claude Code: ±3.27% and ±5.21%), and 78% of kernel gains and 81% of repository gains reproduce. In the incremental ablation, adding a yardstick without a fidelity test actually hurts (10.33×, 66.91%), and the fidelity test recovers it (13.25×, 72.57%); co-discovery adds more (13.76×, 74.43%); the orchestrator adds +1.17× and +6.37 SOL points to reach 14.93× and 80.80%. Removing any single orchestrator action costs 0.63 to 1.01× in repository speedup and 2.3 to 4.4 kernel points.
• Backbone generalization: With Antigravity/Gemini 3.8 Flash, DuoDiscover raises repository speedup from 9.72× to 12.66× and kernel SOL from 58.79% to 73.26%, larger relative gains than with Claude Code/Opus 5, largely closing the gap between backbones; Gemini 3.8 Flash generates 32% more tokens but has 6.7× lower per-token API cost, making the run 5.1× cheaper.

Methodology in Plain English

The starting observation is that big speedups usually come from focused edits to a few small regions of code, and that progress on a small region can often be measured far more cheaply than the whole system. So instead of hand-designing a proxy for each region, the system lets an ordinary LLM coding agent do it: given the task and the starting code, the agent profiles, picks a region worth optimizing, and writes a small scoring script (the yardstick) that returns a score and a correctness flag. The orchestrator reports back how many times faster the yardstick runs than the real evaluation, and the agent decides whether to keep it.

Inside an agent there are two loops. The fast loop edits the chosen region, scores the edit with the yardstick, keeps it if the score improves and reverts otherwise. The slow loop occasionally pays for a real evaluation to confirm a reported gain; no gain is ever accepted without that confirmation. Meanwhile the orchestrator checks whether the yardstick still ranks candidate edits the same way the real objective does, using rank correlation on pairs of states that differ by more than the measured evaluation noise, and invalidates yardsticks that have drifted.

Across agents, the approach is deliberately bottom-up. Nobody splits the problem in advance; each agent picks its own target, and a lightweight LLM orchestrator reads short progress summaries and takes three kinds of action: kill agents that are stuck, start new ones (sometimes seeded with a confirmed solution, or asked to merge several agents' confirmed changes into one place and re-confirm), and send short briefing messages. A run ends when the budget is spent, a target score is reached, or the fleet empties. All methods are compared under identical wall-clock time and agent counts, with scores taken as the median of several re-evaluations of the final candidate and never the best score seen during the run.

Why This Matters

The paper reframes evaluation cost, rather than search cleverness, as the limiting factor for autonomous discovery on real engineering problems, and shows the reframing works from Python repositories down to CPU microarchitecture. It offers a concrete alternative to user-specified proxy evaluators (as in AlphaEvolve's begin/end markers and cheaper checks) by having agents synthesize and then validate their own evaluators during search, which connects the multi-fidelity optimization literature to agentic coding.

Real-world applications:

• LLM serving and inference engines: measurable gains on vLLM multi-objective serving under SLOs and on single-GPU MoE serving in FreeToken, part of which is already merged upstream.
• Databases and compression libraries: 1.38× on SQLite and a 1.16× instruction-ratio improvement on zstd while keeping output byte-identical and passing full test suites.
• Kernel libraries for GPUs and TPUs: first-place rankings on a public SOL-Exec leaderboard and improvements on FA4, JAXBench, and production Pallas kernels.
• Hardware and chip design: prefetcher designs that beat both the strongest prior autonomous system and the best human entry on a public architecture competition, in less search time and with fewer candidates.

Industry relevance: the claimed gains land on heavily optimized production codebases, and the cost accounting (dollars per run, true evaluations consumed, candidate counts, wall-clock days) is presented in the terms engineering teams use to decide whether to deploy such a system.

Future Directions

• When no faithful yardstick exists: the authors state DuoDiscover falls back to standard search over the expensive true objective in that case; how often this happens and how to mitigate it remains open.
• Warmup overhead: building yardsticks and running initial fidelity tests costs upfront, making the approach less advantageous under extremely short evaluation budgets, which bounds where it should be applied.
• Yardstick drift: 41% of repository and 45% of kernel yardsticks are invalidated for drift, so better drift detection or self-repair is a direct target for improvement.
• Fleet scaling and evaluator granularity: the appendix scales the fleet from 1 to 16 agents and compares per-target yardsticks against a single task-wide evaluator, leaving open how fleet size and evaluator scope should be chosen in general.

Target Audience

Researchers and practitioners working on LLM agents, automated program optimization, and multi-fidelity or evolutionary search; performance engineers interested in applying agentic optimization to serving stacks, kernels, databases, or architecture simulators; and evaluation researchers concerned with proxy metrics, reward hacking, and benchmark cost. Readers wanting implementation-level detail will need the appendices, which are only partially represented in the excerpted content.

Authors’ abstract

Evaluation has been central to advances in AI, and it is equally critical to autonomous discovery. Yet scaling autonomous discovery to real-world engineering challenges faces a fundamental bottleneck: the cost of evaluating progress. In real settings, a single true evaluation, such as a full benchmark or serving trace, can take hours, so a search can afford only a handful of true evaluations. We present DuoDiscover, a search system that jointly discovers what to optimize and how to evaluate progress efficiently, through three mechanisms. First, each worker in a fleet of coding agents co-discovers an optimization target and a fast, tailored proxy evaluator, which we call a yardstick. The agent then hill-climbs the yardstick rather than the true evaluation. Second, since search can exploit gaps between a yardstick and the true objective, fidelity tests continually recheck each yardstick's ranking and invalidate any that drift. Third, the fleet is bottom-up, with no central decomposition of the problem. A lightweight orchestrator spawns and terminates agents, briefs running ones, and seeds new agents with others' confirmed work. DuoDiscover enables autonomous optimization from software to chip architecture. We demonstrate DuoDiscover on two repository-level benchmarks and four GPU and TPU kernel suites covering hundreds of kernels. On SWE-fficiency, it doubles the average speedup of human experts. On SOL-Exec, it ranks first on 121 of 235 kernels and in the top three on 212. On the heavily optimized vLLM and SQLite engines, DuoDiscover achieves 1.92$\times$ and 1.38$\times$ gains over the base code. On a CPU microarchitecture optimization task with evaluations of up to 28 hours, DuoDiscover surpasses both AlphaEvolve-based ArchAgent v2 and the best human design, with 1.65$\times$ ArchAgent v2's gain in 16.7 days versus more than 40 days, advancing the frontier of autonomous chip design.