Strategist: Evolving Reasoning with Agents

AI summary

Overview

Research area: Automated design of LLM agent systems — specifically, search over inference strategies (the reasoning procedure that turns model calls into an answer), plus cross-task knowledge accumulation in a reusable code library.

Technical level: Advanced. The paper assumes familiarity with agent scaffolds, prompting baselines (chain-of-thought, self-consistency, tree-of-thoughts, Reflexion), LLM-driven program search, and evaluation on standard reasoning benchmarks.

Scope (one sentence): The paper introduces a meta-agent, Strategist, that writes, executes, and revises inference strategies as composable executable "blueprints," stores every attempt in a shared cross-task library, and is evaluated on 40 benchmarks against 18 baselines and 5 prior automated-design frameworks.

What This Paper Is About

An AI agent system has two halves: the model, and the inference strategy that decides how the model is called, in what order, at what cost tier, and how outputs are checked. The second half is normally written by hand, fixed per system, and never improves on its own — and prior automated methods either search within a fixed component set, or restart from scratch on every new task.

Strategist's goal is to make strategy design cumulative: an agent that designs strategies automatically, tests them on real samples, and deposits every design into a shared library so later tasks begin from what earlier tasks discovered. The models stay frozen; only the strategies change.

Key Contributions

1. The Strategy Blueprint. A representation of an inference strategy as a Python program capturing the structure of the reasoning flow (which models to call, in what order, at which tier) while omitting task-specific prompts. Every blueprint exposes one shared interface, execute(llm, task) -> str, and is built from two foundations — chat and agent — so any blueprint can be used as a component inside another.

2. A growing, protocol-governed Blueprint Library. Blueprints are stored with measured performance and experience notes. Four protocols keep the library composable: a shared interface, no forward calls (a node may only call nodes created before it), no cycles, and no duplicates. Composition depth grows naturally, forming a directed graph of strategies.

3. Strategist, a four-stage meta-agent loop. Given a task and a small development set, the meta-agent Retrieve-s (hybrid TF–IDF plus embedding search over the library, plus "adjacent inspiration" from other domains), Propose-s K = 5 candidates per iteration (each labeled new, composed, or reuse), Evaluate-s them on real samples, and Accumulate-s the underlying blueprints back into the library with measurement cards.

4. Intra-strategy multi-tier routing. Strategies may route each step across three model tiers (fast, balanced, complex), each in two modes (nothink, thinking), so a single strategy can spend capacity only where it changes the answer.

Main Findings

• Main result on 30 benchmarks: Strategist wins on 24 of 30, ties on 2 (MBPP+, MinervaM), trades accuracy for cost on 2 (MedQA, AIME 2024) and loses on 2 (HotpotQA and COPA, at equal accuracy but 9% and 6% higher cost). Average accuracy goes from 74.0% to 74.7% while weighted cost drops 39% (mean per-benchmark cost ratio 0.61×). It cuts cost by more than 2% on 26 of 30.

• Largest joint accuracy-and-cost gains: SQuAD v2 (73.2% → 79.1% at 0.23× cost), TruthfulQA (+4.5 points), 2WikiMQA (+4.0 at 0.34×), Amazon-Rec (+4.0), SAT-Math (+3.6 at 0.41×), LSAT-AR (+3.4), LSAT-RC (+3.0 at 0.50×).

• Near-ceiling benchmarks: where models are close to their ceiling (MMLU, GSM8K, ASDiv, SciQ, MMLU-STEM-Hard), Strategist holds accuracy within 2 points of the baseline and cuts cost by 66–89%.

• One-shot retrieval works: the 30-benchmark run grew the library to 1,865 blueprints. On 10 held-out benchmarks, a single retrieval with no search (top-5 blueprints instantiated, measured on up to 300 development questions, one chosen) stays within 2 points of the baseline on all 10, beats it on 8 (77.8% → 81.4% on average) and wins on 6. Per-benchmark average cost is 0.77× the baseline's excluding BoolQ and SIQA, whose very cheap single-call baselines push those ratios to 32× and 4.5×.

• Cross-model transfer: the same blueprint code and prompts, moved unchanged to GPT-5 (nano, mini, full) and Gemini-2.5 (flash-lite, flash), win on 7 of 10 (GPT-5) and 8 of 10 (Gemini) benchmarks; aggregate cost in USD drops 35% and 40%. Part of the GPT-5 margin comes from baseline workflows that break on the new backend (self_refine and multi_agent_debate return malformed answers) while discovered strategies run unchanged.

• Library ablation (fixed 10-benchmark set): full system 78.7%; same 10-iteration loop with the library disabled 65.4% at 41.4k cost (13.3 points lower, and the search degenerates into narrow repetition without inspiration); zero-shot, one iteration, no library 39.4% with three crashed runs counted as zero (56.2% over the seven that ran, 39.3 points below full).

• Cross-framework comparison: against AFlow (67.0%), ADAS (65.9%), EvoFlow (64.2%), StrategyLLM (67.3%) and AgentSquare (65.1%) — all sharing the same meta-LLM and executor — Strategist wins 8 of 10 benchmarks at 78.7% mean accuracy.

• Limited overfitting of proposals, but selection noise: across the 10-iteration loop the training–hidden gap stays between 1.2 and 2.5 points and does not widen. However, the final strategy is selected on the hidden split: the pick is 4.3 points above baseline on hidden but 0.7 on test (the baseline itself moves by 0.5). The test split, on which nothing is chosen, is reported as the only unbiased estimate.

• What a discovered strategy looks like: the Amazon-Rec example decomposes the task into profile building, signal extraction, a conditional judge, merge, and audit — routing only users with shifting preferences into an expensive multi-agent debate.

Methodology in Plain English

The setup separates two actors. The task models (Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, each with thinking off or on, giving six configurations) actually answer benchmark questions. The strategist is GPT-6, a separate frozen LLM accessed through the Codex CLI that writes strategies but never answers questions.

A strategy is not a prompt; it is a small Python program that says "generate three candidate answers, review each, then majority-vote" — the skeleton of a reasoning procedure. Task-specific prompts get filled in afterward to produce a runnable strategy. Because every blueprint shares one interface, a newly written blueprint can simply call an older one, and depth accumulates.

Each benchmark is split into four parts: a training set (strategist sees scores and full agent traces), a validation set (scores only, to discourage fitting to the trace signal), a hidden set (used only to pick the winning strategy across iterations), and a test set held out entirely from the search. Each of training, validation, and hidden holds up to 100 samples; test is a fixed 200 samples.

The comparison point is deliberately conservative. Each of 18 baselines runs at all six model configurations, giving an 18 × 6 = 108-cell matrix per benchmark; the baseline (BL) is whichever cell is most accurate on the training split, and its test score is reported — mimicking deployment, where you pick a method on the sample you have and then apply it to unseen data. A strategy counts as a win only if it is better on accuracy by more than 2 points or on cost by more than 2%, and worse on neither. Cost is token usage weighted by tier (fast 1×, balanced 2.83×, complex 6.67×, matching the 0.6B/1.7B/4B parameter ratio), since FLOPs scale linearly with parameter count.

Every candidate runs through the same executor, answer extractor, and tool sandbox (OpenHands TerminalTool + FileEditorTool, no internet) as every baseline, so differences reflect strategy choice rather than scoring asymmetry.

Why This Matters

Impact on research. The paper reframes the "strategy layer" as a learnable artifact rather than a hand-tuned constant. It shows that a search procedure over executable strategies can accumulate transferable structure across 30 heterogeneous benchmarks, and that a strategy found on one model family (Qwen3) carries over to GPT-5 and Gemini-2.5 without edits — evidence that the search captures task structure rather than executor quirks. It also introduces a clean evaluation discipline (training/validation/hidden/test with the baseline chosen on training) that exposes selection effects other comparisons can hide.

Real-world applications (illustrative, drawn from the benchmark categories):

• Customer-support and enterprise QA pipelines, where routing easy inputs to a cheap tier and escalating only ambiguous ones cuts inference spend (the paper reports 66–89% cost cuts on near-ceiling tasks).
• Multi-hop document analysis (HotpotQA-, 2WikiMQA-, MuSiQue-style workflows) for research and legal review.
• Recommendation and ranking stages, as in the Amazon-Rec strategy with conditional debate for users with shifting preferences.
• Coding assistants and tool-using agents, where generated strategies invoke an executor and a sandbox as part of the reasoning plan.

Industry relevance. The cost axis is the headline: matching the strongest of 108 baseline cells at 39% lower weighted cost, and 35–40% lower aggregate USD cost after transfer to GPT-5 and Gemini-2.5, speaks to the dominant operational expense of agent deployments. The output is plain, readable, editable Python, which matters for auditing and maintenance; and the fact that the approach requires no training — only frozen models plus a search loop — makes it deployable without fine-tuning infrastructure. The library is also the paper's business argument: each task solved leaves the next task a better starting point, so marginal cost of new tasks falls.

Future Directions

1. Reducing selection noise. The gap between the hidden-set pick (4.3 points above baseline) and its test performance (0.7 points) is large relative to the effect size. The paper reports this openly but does not propose a fix; better selection rules or confidence-aware choice among candidates are open.

2. Library governance at scale. The library reached 1,865 blueprints. The paper enforces shared interface, no forward calls, no cycles, and no duplicates, but does not study retrieval quality, staleness, or growth limits as the library continues to expand across many more tasks.

3. Closing the remaining losses. MedQA and AIME 2024 are cheaper (0.44× and 0.77×) but 4–5 points less accurate; HotpotQA and COPA match accuracy at 9% and 6% higher cost. The paper does not analyze what in the search causes these specific failures.

4. Untested transfer and generalisation surface. Cross-model transfer is measured on 10 benchmarks with GPT-5 and Gemini-2.5 only, and one-shot retrieval on 10 held-out benchmarks. Transfer to genuinely different task distributions, to open-weight executors outside the Qwen3 family, and to much smaller development sets is not reported. The paper's conclusion frames freeform strategy design over an accumulating library as "a step toward inference systems that get better the more they are used," but specifies no further agenda.

Target Audience

Researchers and engineers working on LLM agents, automated agentic-system design (AFlow/ADAS/EvoFlow-style work), and inference-time compute optimization. It is most useful to readers who already know the prompting and multi-agent baselines it compares against and who care about cost-aware deployment. Practitioners building production agent pipelines will find the blueprint representation, the multi-tier routing idea, and the cost methodology directly reusable; ML systems researchers will care about the cumulative-library framing and the train/validation/hidden/test evaluation discipline. Readers new to agents will find the introduction and the Amazon-Rec case study accessible, but the results and ablation sections assume background.

Authors’ abstract

A model is only half of an AI agent system. The other half is the inference strategy: the reasoning procedure that turns model calls into a response. Today that second half is written by hand, and the experience of building one strategy never carries to the next. We introduce Strategist, a meta-agent that evolves inference strategies across tasks. Each strategy is written as a blueprint,executable code that other strategies can reuse, and together the blueprints form a library. Given a task and a small development set, Strategist retrieves related blueprints, writes candidate strategies, tests them on real samples, and revises them from the results. Every candidate then enters the library, so later tasks build on earlier discoveries. On 30 benchmarks, Strategist beats the strongest baseline on 24 and matches its average accuracy (74.7% vs.74.0%) at 39% lower inference cost. Its strategies also transfer to unseen tasks and other models without further search. The part of an AI system that is not improved on its own now does.