ProgramGym: Scaling Training Environments for Long-Horizon Program Reproduction Agents

AI summary

Overview

Research area: Software engineering agents / LLM agent training environments (cs.SE), specifically long-horizon "program reproduction from scratch."

Technical level: Advanced. The paper assumes familiarity with coding-agent scaffolds, sandboxed execution environments, rejection-sampling SFT pipelines, and behavioral (black-box) test generation.

Scope: The paper presents ProgramGym — a framework combining Program-Env (a pipeline that turns open-source repositories into verifiable training environments) and Program-Agent (a role-isolated analysis/implement/review workflow) — and evaluates both on the ProgramBench program-reproduction benchmark.

What This Paper Is About

Most coding agents today either patch bugs inside an existing repository (issue resolution) or implement a repository from a written requirements document. ProgramGym targets a harder setting: rebuilding a program's externally observable behavior when the agent is given only usage documentation and an execute-only reference binary, with no source code and no network. The core problem is that no scalable, verifiable training environments exist for this setting, and existing single-session agent workflows cannot sustain the long-horizon exploration-and-implementation cycle it requires.

Key Contributions

1. Program-Env — a feedback-driven pipeline that builds a reference executable and network-isolated runtime sandbox from a repository and constructs black-box behavioral test suites, using outer loops that repair failed builds and iteratively strengthen tests.
2. Program-Agent — a role-isolated workflow of Analysis, Implement, and Review stages that alternate across independent sessions, combined with revision management providing compile-gated checkpoints and rollback-based failure recovery.
3. A scaled environment corpus — 1,648 training environments spanning C, C++, Go, and Rust, each providing a reference executable, runtime sandbox, usage documentation, and validated black-box tests.
4. An empirical evaluation on ProgramBench showing that Program-Env training (34.87% to 42.84%) and Program-Agent (raising base and SFT models to 55.71% and 56.25%) both improve program reproduction, plus analyses of environment construction, iteration budget, and agent behavior.

Main Findings

• SFT on Program-Env trajectories helps: Holding the scaffold fixed at Claude Code, fine-tuning Qwen3.8-27B on collected trajectories produces Qwen3.8-27B-SFT, which reaches 42.84% Tests Passed versus 34.87% for the base model (Δ = +7.97). This also exceeds the paper's reported GPT 5.4 result of 37.70% under mini-SWE-agent.
• Program-Agent raises the ceiling on both models: With the same weights, Program-Agent lifts Qwen3.8-27B from 34.87% to 55.71% (Δ = +20.84) and Qwen3.8-27B-SFT from 42.84% to 56.25% (Δ = +21.38). Both exceed the paper's reported Claude Opus 4.7 (50.90%) and Gemini 3.5 Flash (53.60%) under mini-SWE-agent.
• A third backbone is also evaluated: DeepSeek-V4-Flash-0731 goes from 57.68% under Claude Code to 58.97% under Program-Agent (Δ = +1.29), with Almost Resolved rising from 4.50% to 5.0%.
• Full reproduction remains unsolved: Tests Passed can exceed 50% while Almost Resolved and Resolved stay near zero. Flagship models under mini-SWE-agent remain at 0.00% Resolved, and after SFT, Almost Resolved falls from 3.50% to 1.50% under Claude Code.
• Test-time scaling with iteration budget: For Qwen3.8-27B-SFT with Program-Agent, Tests Passed rises from 43.49% at K=1 to 54.13% at K=3 and 56.25% at K=5. Five iterations are described as comparable to DeepSeek-V4-Flash-0731 (57.68%), and most of the gain arrives by K=3.
• Environment scale and quality: Program-Env constructs 1,648 tasks from 30,459 collected candidate repositories, at a total cost of 136.77B tokens (about 82.99M tokens per environment on average). Test construction used 5,918 subagents and yielded 665,512 validated test cases, with mean coverage of 72.99% (median 81.63%) and an overall mean of 404 test cases per repository.
• Coverage and interaction profile: Among 5,918 subagents, interaction categories break down as I/O 5,873 (99.24%), Arguments 4,901 (82.82%), Help 4,070 (68.77%), Configuration 2,388 (40.35%), Subcommands 1,305 (22.05%), and TUI 716 (12.10%).
• Difficulty skew: Under the paper's complexity score, the 1,648 tasks split into 1,492 easy, 151 medium, and 5 hard. The paper states this skew is intentional and attributes it partly to cost: processing the full candidate pool was beyond budget, and the 30,459-to-1,648 gap also reflects filtering out repositories that fail to build on Linux, require a display or input devices, or exceed the 3,600-second limit for compilation or instrumentation.
• Behavioral shift: Comparing Qwen3.8-27B and Qwen3.8-27B-SFT under both scaffolds, both models mainly use execute and probe under Claude Code, while Program-Agent increases read and git actions, most after Program-Env training.

Methodology in Plain English

The work has two halves that mirror the two challenges the authors identify.

Building environments (Program-Env). The team started with a stratified GitHub search across C, C++, Go, and Rust, partitioning queries by language and star-count buckets to work around the API's result limit. They validated metadata, removed duplicates, required a non-empty SPDX license identifier, excluded repositories already used by existing code benchmarks, locked each repository to a fixed commit, and screened for build-configuration markers (Cargo.toml, go.mod, CMakeLists.txt, Makefile, meson.build, and package.json as auxiliary evidence). This produced 30,459 candidate repositories.

For each repository, an agent harness (Claude Code) generates a download.sh for online dependency preparation and a compile.sh for offline compilation. An outer loop validates the build and feeds diagnostics back for repair, and only proceeds if both scripts succeed and the executable passes a loadability check. The validated executable becomes the reference binary.

For tests, a dispatcher splits the source into functional modules and assigns each to a parallel subagent. Each subagent iterates: it writes tests, receives feedback documents, and revises. Feedback has three parts — a Summary of what is done and unsolved, Program analysis locating uncovered functionality (dynamic analysis for gaps, static analysis to trace them to public entry points), and Validation consisting of four checks: a pass check (tests that fail on the reference itself), a double check (inconsistent outcomes across repeated runs), a dummy check (tests that accept a dummy implementation), and a weak-assert check (insufficient assertions). Subagents stop when a module reaches the functionality coverage threshold or a round limit; merged suites are re-validated and only tests passing all checks are kept.

Solving tasks (Program-Agent). The agent runs three stages in separate sessions so memory from one stage does not contaminate another. The Analysis stage explores documentation and the reference program without editing the workspace and freezes behavioral evidence as a read-only artifact. The Implement stage works only from that frozen artifact (and cannot access the reference program), delivering a buildable baseline first and then resolving verified issues. The Review stage compares the official reference with the immutable candidate and converts discrepancies into revision requirements under read-only access. Each round is gated on an independent compile of a clean snapshot; successful snapshots are archived as revisions and used as rollback checkpoints if a stage times out, misses artifacts, or fails compilation.

Evaluation. Models were fine-tuned with the ms-swift framework, then evaluated on ProgramBench using its metrics: % Tests Passed (macro-average of per-task pass rates), % Almost Resolved (tasks with at least 95% of tests passing), and % Resolved (tasks passing all tests without a cheating flag). Each task ran in an isolated container with 8 CPU cores, 16 GB of RAM, and 30 GB of storage under an 8-hour wall-clock limit. Because the task often exceeds a 256K maximum context window, Claude Code was chosen as the single-session baseline over mini-SWE-agent, following the ProgramBench protocol of GLM-5.2. Program-Agent ran for up to K=5 iterations.

Why This Matters

Impact on research. The paper shifts the training-environment question from "patch this codebase" to "figure out what the program should do and design it yourself." It argues that existing agent-training resources such as SWE-Gym and SWE-smith do not supply the behavioral supervision that from-scratch reproduction needs, and it distinguishes itself from MindForge, which the authors say is restricted to command-line programs and does not filter trajectories by recovered program behavior. Notably, the paper states it could not report a head-to-head comparison with MindForge because MindForge has not released its environments or trajectories — a candid gap. The paper also notes that high Tests Passed can coexist with near-zero Resolved, which is a useful calibration for how claims about coding agents should be read.

Real-world applications:

• Producing training data for coding agents that must reverse-engineer behavior from a binary and documentation, a common situation when source is lost, restricted, or unavailable.
• Automated compatibility reimplementation — rebuilding a tool's observable behavior in a different language or runtime.
• Generating executable, verifiable regression suites for legacy software directly from a validated reference build.
• Bootstrapping agents for greenfield development where requirements must be discovered rather than handed over.

Industry relevance. The architecture is directly relevant to teams building coding-agent products, since it separates two independent levers: the environment/training data and the agent scaffold. The paper's own results illustrate why both matter — training alone moved Qwen3.8-27B by 7.97 points, while the workflow change moved the same weights by roughly 21 points. The infrastructure profile (8 CPU cores, 16 GB RAM, 30 GB storage, 8-hour limit per task) and the reported cost of 136.77B tokens for 1,648 environments give a concrete sense of the resources required.

Future Directions

• Closing the "Resolved" gap. Tests Passed above 50% with Resolved at or near 0.00% indicates whole-program reproduction remains open, and the post-SFT drop in Almost Resolved under Claude Code (3.50% to 1.50%) suggests the training signal and the hardest tasks are not yet aligned.
• Scaling to medium and hard repositories. The paper states that as model capability grows, the same pipeline can be applied to medium and hard repositories — only 5 of the 1,648 environments are currently hard.
• Constructing environments from the remaining repository pool. The pipeline is described as fully automated and budget-limited rather than capability-limited, leaving roughly 30,459 candidates largely untapped; the authors invite the community to build further environments.
• Better coverage instrumentation and reporting. The paper explains it could not report a mean coverage curve over iterations because subagents terminate at different rounds and coverage is only aggregated at the repository level after all branches finish, and it presents case studies instead.

Target Audience

Researchers and engineers working on coding agents, agent training environments, and LLM post-training pipelines will get the most from this paper. It is also relevant to benchmark designers interested in behavioral verification rather than issue-resolution evaluation, and to practitioners who need to judge what "Tests Passed" actually means relative to fully resolved tasks. Readers without a background in agent scaffolds, sandboxed builds, and SFT data collection will find the paper's infrastructure details dense but its central argument — that discovering requirements is a different and harder task than implementing them — accessible.

Authors’ abstract

Coding agents are increasingly used to build programs from scratch as software engineers do rather than to resolve issues within a repository or to generate a repository from a given requirements document. However, scalable verifiable training environments that force agents to discover what is required and to decide how the system should be designed remain scarce. Moreover, existing agent workflows also cannot sustain long-horizon development in which the agent explores the reference program before implementation. To address these challenges, we introduce ProgramGym, a framework consisting of Program-Env and Program-Agent for scalable training environments of long-horizon program reproduction agents. Specifically, Program-Env constructs verifiable training environments from repositories through a feedback-driven loop that repairs runtime sandbox builds and the quality of black-box tests. Using this novel pipeline, we obtain 1,648 training environments spanning C, C++, Go, and Rust. Program-Agent combines a role-isolated workflow of analysis, implementation, and review with revision management for compilation checks and failure recovery. Experiments on ProgramBench show that both Program-Env and Program-Agent improve performance on program reproduction. Fine-tuning on trajectories with these constructed environments raises the performance of Qwen3.8-27B from 34.87% to 42.84%. Program-Agent further raises the base and SFT models to 55.71% and 56.25%, both exceeding some flagship models. We will release the ProgramGym to support further research on coding agents that build programs from scratch.