KAPSO: A Knowledge-grounded framework for Autonomous Program Synthesis and Optimization

AI summary

Overview

Research area: Autonomous program synthesis, LLM-based coding agents, and execution-grounded black-box optimization (with applications to machine learning engineering and competitive heuristic programming).

Technical level: Advanced. The paper includes formal notation (evaluator contracts, stochastic aggregation operators, typed knowledge graphs) alongside a full system architecture, so readers will get the most from it with background in agentic AI systems and software engineering infrastructure.

Scope in one sentence: The paper introduces KAPSO, a modular framework that turns program synthesis into one operator inside a long-horizon, evaluator-driven optimization loop, and reports end-to-end results on the MLE-Bench and ALE-Bench benchmarks.

What This Paper Is About

LLM coding agents can write code, but they tend to fail over long-horizon tasks: they lose experimental state between iterations, repeat the same integration errors, and fail to reuse engineering expertise that already exists in repositories, documentation, and prior attempts. KAPSO addresses this by treating code synthesis not as the endpoint but as a single step inside a repeated loop of ideation, code synthesis and editing, execution, evaluation, and learning, where progress is defined by what an evaluator measures. The goal is a system that reliably improves a runnable artifact toward a measurable objective while preserving provenance and reusing hard-won lessons across attempts.

Key Contributions

1. An end-to-end framework for evaluator-grounded program optimization that improves runnable artifacts through iterative ideation, code synthesis/editing, execution, evaluation, and learning, with pluggable evaluators and knowledge sources.
2. A git-native experimentation engine that represents each attempt as an isolated, reproducible branch with explicit provenance, persisting diffs, commit identifiers, evaluator configuration (including rollout count K), run logs, structured diagnostics, and evaluator-produced artifacts.
3. A knowledge acquisition and representation pipeline hosted in MediaWiki that converts heterogeneous sources into a typed, workflow-oriented knowledge base with four page types (Principle, Implementation, Environment, Heuristic) and typed edges such as IMPLEMENTED_BY, USES_HEURISTIC, and REQUIRES_ENV, indexed into a graph store (default Neo4j) and vector store (default Weaviate).
4. A cognitive memory system that combines knowledge retrieval with episodic learning from experiment traces (run logs, diffs, evaluator feedback), including error-recovery augmentation (ERA) and a retry/pivot/complete controller policy, to reduce repeated failures and accelerate iteration.
5. A released knowledge package consisting of a MediaWiki dump, Neo4j and Weaviate snapshots, and Docker-based deployment scripts, populated from over 2,000 widely used data and ML repositories, plus a manifest of the repository corpus (URLs, commit identifiers, selection criteria).

Main Findings

• MLE-Bench advantage grows with task difficulty: Leeroo scored 68.18 ± 2.62 on Low-difficulty tasks (tied with R&D-Agent), 44.74 ± 1.52 on Medium, 40.00 ± 0.00 on Hard, and 50.67 ± 1.33 overall. R&D-Agent, the strongest open-source baseline shown, scored 68.18 ± 2.62 Low, 21.05 ± 1.52 Medium, 22.22 ± 2.22 Hard, and 35.11 ± 0.44 overall.
• Other MLE-Bench baselines were lower: AIRA-dojo reached 55.00 ± 1.47 Low, 21.97 ± 1.17 Medium, 21.67 ± 1.07 Hard, 31.60 ± 0.82 All; ML-Master reached 48.48 ± 1.52 Low, 20.18 ± 2.32 Medium, 24.44 ± 2.22 Hard, 29.33 ± 0.77 All; AIDE reached 35.91 ± 1.86 Low, 8.45 ± 0.43 Medium, 11.67 ± 1.27 Hard, 17.12 ± 0.61 All.
• MLE-Bench protocol: Solutions run in debug mode (python main.py --debug) under a strict runtime cap, then in full mode (python main.py) with a longer budget; the run stops after 24 hours, at a $200 maximum budget, or early if any medal is achieved.
• ALE-Bench best final performance, lower cost: Leeroo achieved a final performance of 1909.4 with a rank percentile of 6.1% at a total cost of $914.8, versus ALE-Agent at 1879.3, 6.8%, and $1003.3.
• Large margins on individual AHC competitions: Leeroo reportedly surpassed ALE in most AHC competitions, for example ahc016 (2022 vs. 1457) and ahc026 (2040 vs. 1965), while maintaining comparable rank percentiles.
• Stated limitation: The authors note that the relatively low number of competitions in ALE-Bench may introduce noise; they cite competition ahc039, where ALE-Agent achieves a notably high score that is not consistently reflected in other similar short competitions, and call for larger competition sets and multiple seeds.
• Not reported in the available content: The full ALE-Bench results tables (including per-competition private absolute scores and the complete model column) are truncated, so the specific LLM used for evaluation is not fully stated. Ablations of the three components are listed as a contribution but their results are not shown in the provided content.

Methodology in Plain English

KAPSO is built around a simple contract: give it a natural-language goal, optional knowledge sources, and an evaluator, and it returns a runnable artifact that it then improves by measurement. Users interact through four operations — evolve (run the optimization loop), deploy (package a selected solution into a runtime), learn (ingest repositories, playbooks, or benchmark artifacts into the knowledge base), and research (find external papers, docs, and candidate repositories).

The system separates a knowledge plane from an execution plane. The knowledge plane collects heterogeneous material, hosts it in MediaWiki for human curation, and indexes it into graph and vector stores. The execution plane is anchored by an explicit evaluator boundary: a ProblemHandler defines how the artifact is run, what outputs it must produce, and how quality is measured.

Internally, an OrchestratorAgent composes four swappable subsystems: a SearchStrategy (linear or tree-based, including an LLM-steered tree search that prunes, expands, and selects nodes), a ContextManager that renders the prompt context, a KnowledgeSearch that returns structured knowledge packets with provenance, and a CodingAgent that applies edits. Each iteration builds context from evaluator state, retrieved knowledge, episodic memory, and experiment history, then runs isolated experiments.

Every experiment is a git branch created from a parent branch. The session applies edits, runs the evaluator, commits code changes and run outputs, and attempts to push the branch back into the local ExperimentWorkspace, which makes new branches immediately available as parents for later experiments. The cognitive layer sits on top: after each experiment it extracts a reusable lesson (a fix pattern after a failure, or a best-practice insight after a success), stores it in a vector database with a JSON fallback, and retrieves relevant lessons for the next iteration. A controller then decides whether to retry, pivot to alternative knowledge, or complete — pivoting re-runs retrieval while excluding the currently selected seed repository.

Why This Matters

Impact on research. The paper reframes program synthesis as an operator inside evaluator-grounded optimization rather than a standalone generation task, and provides formal objects (evaluator contract, stochastic aggregation, typed knowledge graph, experiment history) that make agentic coding systems easier to compare and analyze. The released knowledge package with reproducible infrastructure — a MediaWiki dump, Neo4j and Weaviate snapshots, and Docker deployment scripts — is a concrete contribution toward reproducibility in this area.

Real-world applications:

• Automated machine learning engineering for Kaggle-style competitions, where an agent must handle data contracts, training scripts, submission formatting, and iterative tuning.
• Competitive heuristic optimization, producing C++ solutions under strict runtime limits for contests like AtCoder Heuristic Contests.
• Internal engineering copilots that reuse an organization's playbooks, repositories, and past incidents to avoid repeating the same integration failures.
• Deployment pipelines that package an optimized solution into a stable runtime (local, Docker, Modal, BentoML, or LangGraph Platform) behind a single run() interface.

Industry relevance. The cost comparison on ALE-Bench ($914.8 vs. $1003.3) and the explicit budget stops on MLE-Bench (24 hours or $200) speak directly to the economics of running autonomous coding agents. The modular architecture — pluggable evaluators, retrieval backends, coding agents, and deployment strategies — means teams can adopt parts of the framework without committing to the whole stack.

Future Directions

• Scale the benchmark evidence: The authors themselves call for evaluating on a larger set of ALE-Bench competitions and running with multiple seeds to reduce noise in comparisons.
• Ablate the three components: Contributions list ablations as part of the work, but the provided content does not report them; isolating the value of the git-native engine, the knowledge system, and episodic memory would clarify what drives the gains.
• Extend the knowledge base and its selection criteria: The released corpus covers over 2,000 repositories; expanding it and formalizing the selection criteria could improve seed repository retrieval and transfer across domains.
• Broaden beyond the two instantiated benchmarks: The paper argues the same interfaces generalize by swapping evaluator and knowledge sources, but MLE-Bench and ALE-Bench are the only demonstrated instantiations.

Target Audience

Researchers and engineers working on LLM-based coding agents, agentic search, and automated machine learning; infrastructure engineers interested in reproducible experiment provenance (git-native branching, knowledge graphs, vector retrieval); and practitioners building long-horizon autonomous systems where progress is defined by measurable evaluator outcomes rather than one-shot code generation.

Authors’ abstract

We introduce KAPSO, a modular framework for autonomous program synthesis and optimization. Given a natural language goal and an evaluation method, KAPSO iteratively performs ideation, code synthesis and editing, execution, evaluation, and learning to improve a runnable artifact toward measurable objectives. Rather than treating synthesis as the endpoint, KAPSO uses synthesis as an operator within a long-horizon optimization loop, where progress is defined by evaluator outcomes. KAPSO targets long-horizon failures common in coding agents, including lost experimental state, brittle debugging, and weak reuse of domain expertise, by integrating three tightly coupled components. First, a git-native experimentation engine isolates each attempt as a branch, producing reproducible artifacts and preserving provenance across iterations. Second, a knowledge system ingests heterogeneous sources, including repositories, internal playbooks, and curated external resources such as documentation, scientific papers, and web search results, and organizes them into a structured representation that supports retrieval over workflows, implementations, and environment constraints. Third, a cognitive memory layer coordinates retrieval and maintains an episodic store of reusable lessons distilled from experiment traces (run logs, diffs, and evaluator feedback), reducing repeated error modes and accelerating convergence. We evaluated KAPSO on MLE-Bench (Kaggle-style ML competitions) and ALE-Bench (AtCoder heuristic optimization), and report end-to-end performance. Code Available at: https://github.com/Leeroo-AI/kapso