CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

AI summary

Overview

Research area: Reinforcement learning for agentic coding — specifically, automated construction of RL training environments from open-source source code.

Technical level: Advanced.

Scope: CodeMidas is an agentic pipeline that converts already-implemented functionality in open-source codebases into executable coding RL environments using source code as the only task-specific input, yielding 5,545 verified training tasks which are used to train MiMo-V2.5 with GRPO.

What This Paper Is About

Training coding agents with reinforcement learning needs many diverse tasks paired with trustworthy automatic verifiers, but existing pipelines build those tasks from development records such as issues, pull requests, commits, docs, or pre-existing tests — which caps how much data can be extracted. CodeMidas instead treats the source code itself as the raw material: it finds functionality that already exists, turns it into a task statement plus a stripped-down starting codebase, and synthesizes tests by actually running the original implementation. The goal is to scale RL environment construction beyond the coverage of recorded development history.

Key Contributions

1. An agentic pipeline that uses source code as its only task-specific input. CodeMidas allocates agentic compute to every stage of environment construction: exploring implemented functionality to write behavioral specifications, adapting codebases so the target functionality is missing, constructing execution-grounded tests, and validating tasks through execution checks and repeated rollouts. Unlike comparable pipelines in Table 1, it requires no issues, no pull requests, no commits, no existing tests, and no written description.

2. A large multi-language, multi-domain task dataset. 5,545 training tasks drawn from 3,185 open-source codebases, spanning 23 programming languages and 15 technical domains — more languages than the comparison pipelines listed in Table 1 (SWE-rebench V2: 20; MindForge: 15; SWE-Hub: 11; R2E-Gym, SWE-smith, SWE-Flow, R2E, daVinci-Env: 1 each).

3. Empirical gains across five external benchmarks after GRPO training of MiMo-V2.5, covering issue repair, whole-program construction, code translation, and terminal work.

4. An analysis of task scale versus task quality, plus trajectory-level behavioral analysis, showing that a smaller cleaned dataset beats a larger unfiltered one, and that RL-trained agents explore more and self-verify more diversely — behaviors that also transfer to held-out benchmarks.

Main Findings

• All five external benchmarks improve. Training on CodeMidas raises DeepSWE pass rate from 10.0% to 21.7% (+11.7 points), ProgramBench Almost Solved score from 4.5 to 21.5 (+17), and Terminal-Bench v2.1 pass rate from 63.7% to 72.2% (+8.5). The paper states performance improved on all five benchmarks including SWE-bench Pro and RepoZero C2Rust; the specific numeric gain for RepoZero C2Rust is not reported in the provided content.

• CodeMidas Val tracks improvement during training. On the 200-task held-out set, pass rate rises from 35.0% to 44.7%, staying roughly 8–10 percentage points above the initial rate at evaluated checkpoints from step 40 onward, accompanied by longer trajectories.

• More high-quality tasks produce better results. DeepSWE scores rise from 17.57 to 19.05 to 21.70 for the 1k, 3k, and 5k pools respectively, and CodeMidas Val rises from 41.30 to 43.22 to 44.73. On CodeMidas Val, the 1k setting reaches 41.30 at step 30, the 3k setting reaches 43.22 at step 65, and the full dataset reaches 44.73 at step 70, leading at every evaluated checkpoint from step 40 through step 70.

• Quality beats raw quantity. The full 5k high-quality pool outperforms a vanilla ~8,000-task sample (drawn before filtering and cleaning) by 0.59, 4.59, and 4.49 percentage points on SWE-bench Pro, DeepSWE, and CodeMidas Val. Even the high-quality 3k subset beats the vanilla 8k sample on all three evaluations.

• Agent behavior shifts during RL. Pre-edit read/search calls increase from 27.2 to 40.1, the drafting ratio (fraction of written code fragments already present in preceding reasoning) rises from 0.358 to 0.629, and distinct post-edit verification commands rise from 2.03 to 2.53.

• Self-verification correlates with success. Within the same task and checkpoint on CodeMidas Val, rollouts containing agent-written and executed checks have a mean pass rate 4.2 percentage points higher than those without (95% CI: 1.8–6.6). Exploration showed +0.7 points (95% CI: -1.9 to 3.7) and drafting +1.95 points (-0.04 to 3.96).

• Behavioral changes generalize to held-out benchmarks. From the first to the last three observed checkpoints, SWE-bench Pro exploration goes 23.1 → 35.5 with interaction length 37.3 → 50.1 turns; ProgramBench exploration goes 55.7 → 83.6 while interaction length decreases from 155.1 → 122.8 turns; Terminal-Bench v2.1 exploration goes 11.9 → 16.8 with interaction length 59.2 → 69.5 turns.

• Dataset composition. Python (21.4%), TypeScript (18.3%), and Go (16.2%) dominate, followed by C++ (12.5%) and JavaScript (11.3%); the ten most frequent languages cover 5,445 of 5,545 tasks (98.2%). Systems software (17.4%), web technologies (14.6%), and developer tools (13.6%) together account for 45.6% of tasks, with 18 tasks (0.32%) unlabeled.

• Reference solutions are substantial. Median reference patch size is 142 lines (interquartile range 66–305), and reference patches touch at least two source files in 65.9% of tasks.

Methodology in Plain English

Each CodeMidas task bundles three things: a written statement of required behavior, a containerized development environment, and a hidden executable verifier. The verifier is kept outside the solver's environment and injected only at grading time to produce a binary reward.

The pipeline has four stages:

1. Task design and codebase adaptation. An agent inspects the repository's structure and build metadata to find functionality with public entry points — CLI tools, library functions, or stateful APIs. It identifies the task scope, deletes the core implementation, and adjusts what remains so the codebase still builds and looks like a real project. The deleted implementation is kept aside as a reference solution.

2. Execution-grounded test construction. Tests are derived by actually running the reference solution and recording outcomes, rather than by guessing expected values. Different interface types get different test styles — command executions for CLI tools, input/output cases for pure functions, call sequences for stateful APIs. A review pass strips out assertions that lock in unspecified details (for example, exact exception wording or incidental ordering) but keeps checks the statement requires. A task is rejected if an assertion depends on a private symbol with no behavioral substitute.

3. Environment preparation and execution consistency. A uniform base container is used; an agent installs dependencies and cleans away artifacts that could leak the deleted implementation, including compiled outputs, caches, files left by construction agents, and the original tests for the target functionality. Each task is then verified in six fresh containers: two with the starting codebase must fail, and four with the reference solution must pass.

4. Post-rollout filtering. Three checks run before RL training: an adversarial rollout that hunts for leftover leakage that would let a solver bypass the real work; an agreement check where a coding agent makes four attempts per task and a reviewer compares the trajectories against the statement, verifier, and reference solution to catch false positives and false negatives; and an outcome filter that keeps only tasks where a frontier model produces both successful and failed attempts.

Training uses GRPO with binary execution rewards, batch size 32, 32 rollouts per task, maximum prompt length 8,192 tokens, maximum response length 516,096 tokens, up to 500 turns per rollout, max staleness 8, Adam with learning rate 5×10⁻⁶, β = (0.95, 0.95), ε = 10⁻¹⁵, gradient clipping threshold 1, no weight decay, no warmup, and advantage normalization by standard deviation disabled.

Why This Matters

The paper argues that the supply of coding RL training data has been bottlenecked by dependence on development artifacts — issues, pull requests, commits, docs, existing tests — which only cover what humans happened to record. If source code alone is sufficient, the available pool of extractable tasks grows enormously, since open-source corpora span hundreds of programming languages. The paper also emphasizes verifier reliability over raw volume, showing that a filtered 5k pool beats an unfiltered 8k pool, and documents behavioral evidence of generalization rather than only benchmark deltas.

Real-world applications:

• Automated issue repair in large repositories, where DeepSWE pass rate moved from 10.0% to 21.7%.
• Whole-program construction from a description, where ProgramBench Almost Solved went from 4.5 to 21.5.
• Terminal and shell-heavy operations work, where Terminal-Bench v2.1 pass rate moved from 63.7% to 72.2%.
• Cross-language code translation, evaluated via RepoZero C2Rust, one of the five benchmarks the paper reports gains across.

Industry relevance: the dataset spans 23 languages and 15 domains, so a single training pipeline could plausibly serve organizations working in polyglot codebases rather than single-language shops. Because the environment is containerized and the verifier returns a binary execution reward with no learned reward model, the approach is comparatively straightforward to deploy as a training loop. And because agents trained on these tasks explore codebases more and run more varied self-verification, the resulting behavior is directly useful for long-horizon agentic deployment.

Future Directions

• Pushing scale further. The ablation covers only 1k, 3k, 5k, and an 8k pre-filtering sample. Whether gains keep accruing past 5,545 cleaned tasks, and where returns diminish, is not established.

• Deeper verifier reliability work. The paper cites evidence that expanded tests uncover wrong programs missed by original suites (EvalPlus) and that incorrect patches have passed SWE-bench tests (PatchDiff), then layers leakage filtering, solution agreement review, and rollout outcome filtering on top. How residual false-positive and false-negative rates look after filtering is not quantified in the provided content.

• Understanding which filtering step matters most. The 5k-versus-8k comparison bundles environment cleaning, execution consistency checks, and all three post-rollout filters together. Ablating the individual components would clarify where the value lies.

• Explaining the ProgramBench interaction-length result. On ProgramBench, mean interaction length decreased from 155.1 to 122.8 turns while exploration increased from 55.7 to 83.6 calls — the paper notes greater exploration accompanies shorter overall interactions, but the mechanism is an open question.

• Broadening beyond the measured task types. The evaluation covers five benchmarks; how far these behaviors transfer to domains outside the 15 labeled technical domains remains untested.

Target Audience

Researchers and engineers working on reinforcement learning for coding agents, LLM post-training, or synthetic data generation for software engineering tasks. It is also relevant to teams building verifier or reward infrastructure for agent training, and to practitioners who need to know whether training data derived from source code alone can outperform data derived from issues, commits, tests, or documentation. Readers should have some familiarity with RL training loops (GRPO, rollouts, rewards) and with repository-level software benchmarks to follow the ablation and behavioral analysis sections.

Authors’ abstract

Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich source of such tasks, while existing methods typically rely on development artifacts such as issues and commits, limiting the range of tasks that can be extracted. To better scale RL environments, we present CodeMidas, an agentic pipeline that turns implemented functionality in existing codebases into executable RL environments using source code as its only task-specific input. CodeMidas allocates agentic compute to every stage of environment construction: agents explore implemented functionality to formulate behavioral specifications, construct tests grounded in execution of the original code, and validate and filter candidate tasks through execution checks and repeated solution rollouts. The resulting dataset has 5,545 training tasks from 3,185 open-source codebases spanning 23 programming languages and 15 technical domains. Training MiMo-V2.5 on these tasks with GRPO improves performance on all five diverse benchmarks, covering issue repair (DeepSWE + 11.7%), whole-program construction (ProgramBench +17%), and terminal work (Terminal-Bench v2.1 +8.5%). Ablations show that increasing the number of high-quality training tasks improves performance. Trajectory analysis shows the RL-trained agent demonstrates better behaviors like increasing codebase exploration and more diverse self-verification. These results establish source code as a scalable foundation for constructing RL environments that improve coding agents across diverse software tasks.