CodeEvolve: an open source evolutionary coding agent for algorithmic discovery and optimization

AI summary

Overview

Research area: Automated algorithmic discovery and program synthesis, at the intersection of large language models (LLMs) and evolutionary computation.

Technical level: Intermediate. Readers need familiarity with evolutionary search concepts (populations, crossover, fitness, migration) and with using LLMs as code generators, but the paper is written so that the architecture can be followed without deep mathematics.

Scope in one sentence: The paper presents CodeEvolve, an open-source framework that combines island-based evolutionary search with modular LLM operators to evolve complete programs, and benchmarks it against AlphaEvolve, ThetaEvolve, OpenEvolve, ShinkaEvolve, and Evolution of Heuristics (EoH).

What This Paper Is About

Automated algorithmic discovery systems such as AlphaEvolve can produce state-of-the-art solutions but are closed-source and depend on proprietary frontier models, which limits reproducibility and raises cost. CodeEvolve is an attempt to reproduce that capability in a fully open framework by orchestrating smaller, open-weight models inside an evolutionary search loop. The goal is to find out whether a transparent, modular combination of prompt evolution and program evolution can match or beat reported closed-source results at substantially lower cost.

Key Contributions

1. An open-source framework for end-to-end algorithmic discovery that integrates islands-based evolutionary search with modular LLM orchestration, designed explicitly for transparency and reproducibility. The framework, experimental data, and hyperparameter guidelines are released at the project repository.

2. A comprehensive empirical evaluation on established algorithm-discovery benchmarks, covering reported closed-source baselines (AlphaEvolve, ThetaEvolve) and controlled open-source comparisons (OpenEvolve, ShinkaEvolve) under matched budgets, illustrating quality–cost trade-offs with open-weight models.

3. An extensive ablation and component-level analysis showing that the synergy between CodeEvolve's operators, rather than any single operator in isolation, is the key enabling factor for its strongest results.

4. Three modular evolutionary operators — inspiration-based crossover, meta-prompting exploration, and depth-based exploitation — layered on a CVT-MAP-Elites archive and a weighted LLM ensemble, with practical default hyperparameters that transferred across six LLM backbones without retuning.

Main Findings

• Matching or beating AlphaEvolve: On the AlphaEvolve benchmark suite, CodeEvolve matches or surpasses the reported AlphaEvolve results on 5 of 9 problems, and establishes new best-known results on MinimizeMaxMinDist and CirclePackingSquare (n=32).

• Beating open-source competitors head-to-head: Under matched configurations and evaluation budgets with Qwen3-Coder-30B, CodeEvolve achieves the best solution in 6 of 9 problems and the best mean fitness also in 6 of 9 problems against OpenEvolve and ShinkaEvolve.

• Open-weight models suffice for some problems: With Qwen3-Coder-30B, CodeEvolve surpasses the reported AlphaEvolve score on both CirclePackingSquare instances.

• Order-of-magnitude cost advantage: On CirclePackingSquare (n=26), Qwen3-Coder-30B surpassed AlphaEvolve after roughly 900 model calls at approximately $6, whereas GEMINI-2.5 required approximately 400 calls at about $35.

• Complementary ensembles: Qwen3-Coder-30B achieved the strongest results on the CirclePackingSquare instances and both autocorrelation problems, while GEMINI-2.5 performed best on CirclePackingRect, both HexagonPacking instances, and both MinimizeMaxMinDist instances.

• Weak spot on analytic constructions: The main gap versus AlphaEvolve and ThetaEvolve is on the autocorrelation inequalities. CodeEvolve reached 1.51343 on FirstAutocorrIneq versus AlphaEvolve's 1.50316 and ThetaEvolve's 1.50313, and 0.88110 on SecondAutocorrIneq versus AlphaEvolve's 0.96102 and ThetaEvolve's 0.94690.

• Competitive on heuristic design without retuning: On the EoH suite, CodeEvolve's best FSSP run matches EoH's overall average (0.23 gap) and beats EoH on 6 of 11 Taillard test sets. CodeEvolve significantly outperforms EoH on the out-of-distribution OBPP C500 instances (0.00% versus 2.13% at 1k C500), while EoH leads on C100 (0.61% versus CodeEvolve's best 3.22% at 10k C100). On TSP, CodeEvolve matches EoH on TSP20 (0.000) and is slightly behind on TSP50 (0.001 versus 0.000) and TSP100 (best 0.040 versus 0.025).

• Components interact rather than act alone: In every ablation, the full CodeEvolve configuration produced the best peak performance. On CirclePackingSquare (n=32), it was the only configuration to surpass AlphaEvolve; on n=26, the naive baseline required more than twice as many evaluations to match the full method. Depth-only configurations (iota=0) did not exceed AlphaEvolve, while inspiration-only configurations (k=0, iota=2,3) did; combining both gave the highest sample efficiency.

• Robust defaults: Ring (Cycle) migration topology with migration rate 0.1, initial exploration rate 0.2 with the Plateau Scheduler, CVT-MAP-Elites using fitness and evaluation time as descriptors, maximum ancestor depth k in [3,5], and iota in {2,3} inspirations worked across all benchmarks and the six LLM backbones tested (Qwen3-Coder-30B, GEMINI-2.5, GPT-4.1, Claude Sonnet 4.5, o4-mini, and gpt-oss-20B) without prompt or operator retuning.

Methodology in Plain English

CodeEvolve treats program writing as a search problem. Instead of asking a model once for a solution, it maintains many candidate programs at once, split across several independent "islands" that evolve in parallel and periodically exchange their best individuals (migration), an approach borrowed from island genetic algorithms.

Two things are evolved side by side: the solution programs themselves and the natural-language prompts used to generate them. At each step, the system decides whether to exploit or explore, based on an exploration rate controlled by a scheduler. The Plateau Scheduler watches whether the best-so-far fitness has stopped improving over a sliding window; if progress stalls, it temporarily raises the exploration rate, then anneals back down when improvement resumes.

Three operators do the work:

• Depth exploitation picks a high-performing parent (with probability inversely proportional to its rank) and prompts the LLM with that parent, its parent prompt, and its k nearest ancestors, encouraging incremental refinement rather than wholesale rewrites.
• Meta-prompting exploration samples a solution and a prompt at random, uses an auxiliary LLM to generate an enriched prompt from them, and generates a new solution from that enriched prompt — deliberately ignoring the ancestor chain so new strategies are not constrained by lineage.
• Inspiration-based crossover supplies the LLM with several other solutions as inspiration. Because CodeEvolve uses a diff-based SEARCH/REPLACE editing format on a single parent program, traditional binary crossover is awkward; giving the model multiple inspirations achieves the same intent, with binary crossover as the special case of one inspiration.

A weighted ensemble of LLMs generates the code; a model is sampled according to ensemble weights for each generation task, and users can configure different ensembles for exploration and exploitation. New solutions run in a sandbox with runtime and memory limits; failures get fitness zero and their logs are kept as context for future prompts. A per-island MAP-Elites archive (optionally using centroidal Voronoi tessellations, CVT-MAP-Elites) stores the most fit solution per feature cell, so that inspiration sampling always has diverse material available and premature convergence is reduced.

Experiments were run on AWS SageMaker with a fixed per-run budget in vCPUs and RAM, each candidate evaluated in an isolated sandbox. Comparisons with AlphaEvolve and ThetaEvolve use their reported numbers because AlphaEvolve is closed-source and ThetaEvolve requires large-scale RL infrastructure; comparisons with OpenEvolve and ShinkaEvolve were run under matched configurations and budgets. The EoH comparison matched the total number of LLM calls and evaluation budget, running CodeEvolve with 10 islands: 154 epochs for OBPP (30s timeout), 152 for TSP (60s), and 152 for FSSP (120s), reporting best and mean over three independent runs.

Why This Matters

Impact on research: The paper argues that the barrier to automated algorithmic discovery is not only model capability but orchestration. If open-weight models inside a well-designed evolutionary loop can match reported results from a proprietary frontier ensemble on 5 of 9 benchmark instances — and beat it on both CirclePackingSquare instances at roughly an order of magnitude lower cost — then reproducibility in this subfield becomes far more achievable. The ablation finding that no single operator suffices also shifts attention from inventing new operators toward understanding how existing ones interact.

Real-world applications:

• Packing and layout problems: The CirclePackingSquare, CirclePackingRect, and HexagonPacking benchmarks map onto practical tasks such as arranging components on a circuit board, cutting stock, or packing objects in a container.
• Logistics and scheduling: The EoH suite includes the Traveling Salesman Problem and the Flow Shop Scheduling Problem, both standard models for routing and production scheduling.
• Resource allocation under streaming constraints: The Online Bin Packing Problem models allocating items into the fewest bins when items arrive one at a time, relevant to memory allocation, network buffering, and warehouse operations.
• Cost-constrained scientific computing: The demonstrated cost profile (roughly $6 of Qwen3-Coder-30B calls versus about $35 of GEMINI-2.5 calls on one instance) makes automated search more plausible for teams without frontier-model budgets.

Industry relevance: The framework is released as open source with practical hyperparameter guidelines, and the same defaults transferred across six LLM backbones and six model families. That suggests the orchestration layer is not tied to a single vendor, which matters for organizations that want to swap models as pricing or availability changes. The dependence on open-weight models also reduces reliance on proprietary APIs.

Future Directions

• Specialized handling for analytic problems: CodeEvolve underperforms on the autocorrelation inequality benchmarks, where AlphaEvolve and ThetaEvolve achieve substantially better results. The authors suggest such problems may require highly specialized analytic constructions that are less amenable to combinatorial search, and could benefit from domain-specific adaptations or stronger backbone models.

• Fairer comparison against closed and RL-based systems: Because AlphaEvolve is closed-source and ThetaEvolve requires RL fine-tuning infrastructure beyond the authors' budget, those comparisons rely on reported numbers, and differences in evaluation budgets, parallelism, or implementation details may still affect the conclusions. Closing that gap remains open.

• Broader ablation coverage: A full ablation grid over problems, components, and LLM backbones is combinatorially expensive, so the most detailed ablations are concentrated on Qwen3-Coder-30B. Extending the grid would test whether the interaction effects generalize.

• Hyperparameter transfer to new domains: The framework introduces hyperparameters (migration topology, number of islands, number of inspirations, maximum ancestor depth) that may require calibration on novel domains. Robust defaults are provided, but the authors state they cannot guarantee these transfer to all problem types.

Target Audience

Researchers and practitioners in automated program synthesis, LLM-driven search, and evolutionary computation who want a reproducible, open-source alternative to closed systems like AlphaEvolve. It is also useful for engineers evaluating the cost–performance trade-off between open-weight and proprietary models for code generation, and for graduate students looking for a well-documented baseline and ablation study in the "evolution through large models" line of work. Readers primarily interested in mathematical bounds on autocorrelation inequalities or in RL-based fine-tuning methods will find less directly applicable material.

Authors’ abstract

We introduce CodeEvolve, an open-source framework that couples large language models with island-based evolutionary search for end-to-end algorithmic discovery. CodeEvolve integrates inspiration-based crossover, meta-prompting, and depth-based refinement on top of a CVT-MAP-Elites archive and a weighted LLM ensemble to generate optimized solutions for complex problems. On the AlphaEvolve benchmark suite, CodeEvolve matches or surpasses the reported AlphaEvolve results on 5 of 9 problems and, under matched conditions, outperforms the open-source frameworks OpenEvolve and ShinkaEvolve on 6 of 9. With the open-weight Qwen3-Coder-30B backbone, it surpasses the reported AlphaEvolve score on both CirclePackingSquare instances at roughly an order of magnitude lower cost than a frontier closed-source ensemble, and remains competitive with EoH on heuristic-design tasks without retuning. Ablations show that the interaction between CodeEvolve's components, rather than any single operator, drives these results. We release the framework, experimental data, and practical hyperparameter guidelines at https://github.com/inter-co/science-codeevolve.