Initialization Improves LLM-Driven Discovery

AI summary

Overview

Research area: Machine learning — specifically LLM-driven automated discovery (using LLMs as optimizers to invent algorithms, prompts, and other artifacts), with a focus on inference-time compute allocation and harness design.

Technical level: Intermediate. The paper is readable with a general understanding of LLM prompting and iterative optimization, but the experimental apparatus (15 discovery harnesses, five tasks, diversity metrics) assumes some familiarity with the "LLM-as-optimizer" literature.

Scope: An empirical study of how the population of previously discovered candidates inside an LLM discovery harness shapes eventual discovery success, culminating in a simple compute-stratification method ("initialize, then iterate") that improves results across harnesses, tasks, and LLMs.

What This Paper Is About

LLM-driven discovery systems work by repeatedly prompting a model to mutate and improve a candidate solution, feeding recent candidates back into the prompt. The paper asks how the history of candidates maintained in the prompt affects the quality of what is ultimately discovered — in particular, whether sequentially refining a single seed is really the best use of a fixed token budget. The authors find that sequential harnesses tend to collapse onto narrow, similar outputs ("mode collapse"), that efforts to force diversity give inconsistent gains, but that the quality of the earliest candidates predicts final success — so they propose spending part of the budget on parallel exploration to initialize a strong starting population before switching to sequential refinement.

Key Contributions

1. A modular harness suite. The authors generalize the OPRO framework (Yang et al., 2024) into a suite of 12 "Modular" harnesses by crossing 4 LLM mutation strategies (K Context, Reflect, Differential Evolution, Genetic Algorithm) with 3 parent-sampling strategies (Highest Scoring, Tournament, Roulette Wheel), and compare them against 3 state-of-the-art harnesses: ShinkaEvolve (SE), OpenEvolve (OE), and optimize_anything (OA) — 15 harnesses total.
2. A characterization of brittleness and mode collapse. Across 5 discovery tasks and multiple LLMs, the identity of the best harness changes per (task, LLM) pairing, and sequential trajectories show a measurable decline in diversity over time.
3. A negative result on exploration. Neither state-of-the-art diversity-aware harnesses nor three online diversity interventions (cosine similarity thresholding at s ∈ {0.95, 0.8}, an LLM-judge variant, and deduplication) consistently improve discovery quality, despite demonstrably increasing population diversity.
4. An initialization method plus a budget rule. The paper proposes warm-starting any sequential harness with a high-performing population curated from a parallel exploration phase, and shows that a simple convergence criterion on the parallel pool predicts a near-ideal initialization budget. This yields 2%–21% gains when applied to state-of-the-art harnesses.

Main Findings

• Sequential vs. parallel is task-dependent. Parallel prompting (repeatedly mutating the seed without updating context) beats some but not all Modular harnesses. For TSP, parallel discovery typically underperforms sequential harnesses; for Circle Packing the trend is reversed.
• Mode collapse is a common failure mode. Sequential harnesses consistently produce less diverse discoveries than parallel schemes, and windowed diversity (window size 15, except 6 for Circle Packing) declines over the trajectory. The authors attribute this to the in-context prior being reinforced throughout sequential discovery.
• Exploration does not reliably help. State-of-the-art harnesses and diversity interventions do achieve more diverse populations on average (e.g., in Table 2, Modular's average diversity on CloudCast is 0.1271 ± 0.0195 versus 0.1417 ± 0.0132 with interventions and 0.1680 ± 0.3698 for state-of-the-art harnesses), but this does not translate into better maximum scores.
• State-of-the-art harnesses are not consistently better. In Table 4, Modular versus state-of-the-art averages per task: Can't Be Late −91.94 ± 2.75 vs. −91.32 ± 8.33; CloudCast 0.0098 ± 0.0002 vs. 0.0095 ± 0.0021; TSP −1719 ± 16 vs. −1772 ± 250; Circle Packing 2.44 ± 0.08 vs. 2.51 ± 0.07; Prompt Optimization 0.53 ± 0.03 vs. 0.47 ± 0.20. The best choice varies by setting.
• Early quality predicts final quality. Across 72 Modular harness configurations and 3 state-of-the-art harnesses, the median score of the first 15 discoveries correlates with the maximum score achieved in the trajectory.
• Initialization raises the floor. Spending a fraction r of the token budget on parallel exploration and curating a population of size m = 15 (similarity threshold s = 0.98) outperforms both purely sequential (r = 0) and purely parallel (r = 1) discovery across tasks and LLMs, under the same total budget.
• Diversity filtering during curation should be mild. Greedy selection (s = 1) generally performs well but can underperform minimal-diversity rejection sampling (s = 0.98) when the pool contains too many exact or near-duplicates; pushing diversity further (s < 0.98) lowers average population quality and downstream scores.
• Convergence of the parallel pool predicts the right budget. The point at which parallel discovery quality converges consistently coincides with the best-performing initialization budgets, and the same task-specific criterion generally transfers across LLMs.
• Gains on state-of-the-art harnesses. Explicitly initializing SE, OE, and OA improves downstream discovery by 2%–21% across all (task, LLM) pairings examined, at equal total budget.

Methodology in Plain English

The authors build a controlled experimental sandbox. They take one base recipe — show the model the task and some past attempts, ask it to produce a better attempt, score the attempt, keep it if useful — and vary two knobs: how the model is asked to mutate candidates, and how parent candidates are chosen. Twelve combinations give twelve harnesses that all share the same skeleton, so differences in performance can be attributed to design choices rather than implementation details. They run these on five optimization tasks: CloudCast (multi-cloud data transfer cost), Can't Be Late (spot-instance job scheduling), Circle Packing with 26 circles, Traveling Salesperson with 100 cities, and Prompt Optimization for GSM8K. Optimizer LLMs include Gemini 3.5 Flash, Gemini 3.7 Flash, GPT-OSS (120B), and Llama 3.1 Instruct (8B). Cost is measured in tokens rather than number of evaluations, so schemes that spend more tokens per candidate are not unfairly advantaged. Diversity is measured as one minus the average pairwise cosine similarity of candidate embeddings, using all-MiniLM-L6-v2 for natural-language tasks and nomic-ai/CodeRankEmbed for coding tasks. For the proposed method, they first run a parallel phase to build a pool of candidates, then greedily select a high-performing, sufficiently distinct population of size 15, then hand that population to any sequential harness. A convergence check on the parallel pool decides when to stop exploring and start refining.

Why This Matters

The paper reframes a common assumption: that more exploration or more sophisticated harness engineering is the route to better automated discovery. Instead, the biggest lever identified is where the trajectory starts. This is practically useful because initialization is harness-agnostic — it can be bolted onto existing systems without redesign, and it comes with a rule for how much budget to spend.

Real-world applications:

• Automated algorithm design and code optimization, where candidate solutions can be executed and scored automatically.
• Prompt engineering pipelines that search for instructions maximizing a downstream benchmark score.
• Scientific hypothesis or molecule generation workflows that use an LLM proposer with an external evaluator.
• Scheduling and logistics optimization (the CloudCast and spot-instance scheduling tasks are direct instances).

Industry relevance: Teams deploying LLM-based optimization loops care about token cost. The paper's central claim — that a modest, well-chosen fraction of the budget spent on parallel seeding beats spending everything on either pure parallel sampling or pure sequential refinement — is directly actionable for cost/quality tradeoffs, and the convergence-based stopping rule removes the need for task-specific tuning.

Future Directions

• Quantify parametric knowledge for discovery. The paper explicitly calls for better measurement of what the LLM already knows about long-tail discovery tasks, since parallel discovery depends entirely on parametric knowledge and its ceiling is unknown a priori.
• Study the interaction between parametric and in-context priors. The authors suggest reframing obscure discovery objectives in terms of domains the model is strong in — for example, expressing an algorithm design problem in a popular rather than rare programming language.
• Adaptive exploration. A trajectory starting from poor candidates may benefit from diversity interventions, while one starting strong may be hurt by them; distinguishing these cases automatically is left open.
• Batched parallel discovery. Because fully parallel discovery cannot be monitored for convergence mid-run, the authors suggest batched parallel discovery to enable early convergence detection and timely switching to sequential refinement.
• Homogenization risk. The related work raises the concern that populations of LLMs produce increasingly similar outputs, implying future LLM-generated discoveries could converge — an open societal question the paper does not resolve.

Target Audience

Researchers and practitioners working on LLM-based optimization, automated algorithm or prompt discovery, and inference-time compute scaling. It is especially relevant to engineers building or tuning "discovery harness" pipelines who want an evidence-based default for budget allocation, and to researchers studying diversity and mode collapse in iterative LLM generation. Readers seeking a first introduction to LLM-as-optimizer methods will find the harness taxonomy useful but should expect familiarity with the prior literature.

Authors’ abstract

Large Language Models (LLMs) have been used for novel discovery of algorithms, theorems, drugs, and other tasks through the use of harnesses that prompt an LLM to iteratively optimize an objective. In this work, we study the relationship between the population of previous iterates and eventual discovery success. We generalize past work on harness design to develop a suite of 12 harnesses called 'Modular' and characterize their performance across 5 diverse discovery tasks, finding that discovery success is brittle and sensitive to harness design. We uncover mode collapse, characterized by a dramatic drop in the diversity of iterates, as a common failure mode. We find that popular state-of-the-art harnesses and diversity-inducing harness interventions, which aim to prolong this collapse, yield inconsistent gains. Our results instead uncover that the performance of early discoveries is predictive of eventual success. We therefore propose a universally applicable intervention that performs an initial stage of parallel exploration in order to initialize subsequent iterative optimization. Our method provides consistent gains across many harnesses and target applications, confirming the importance of initialization in LLM-driven discovery.