ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments

AI summary

Overview

Research area: Natural Language Processing / AI for Science — specifically agent training environments, supervised fine-tuning, and reinforcement learning with verifiable rewards derived from scientific software.

Technical level: Intermediate. The conceptual framing (turning repositories into graded environments) is accessible to a general technical reader, but the RL section assumes familiarity with group-relative policy optimization, importance sampling, and long-horizon credit assignment.

Scope in one sentence: The paper introduces ScienceIDE, an infrastructure that converts scientific code repositories into executable, verifiable environments used to generate training tasks, evaluate scientific agents, and fine-tune a family of models (PhAI-IDE-4B/9B/72B) that improve both on held-out scientific repair and on general code, reasoning, and knowledge benchmarks.

What This Paper Is About

Scientific repositories contain decades of executable knowledge, but fragmented toolchains, undocumented domain conventions, and physics-based correctness criteria make it hard to turn that code into usable training experience for AI agents. The authors call this the scientific experience bottleneck: papers, repos, and datasets do not automatically become trainable environments. ScienceIDE is the infrastructure proposed to close that gap, packaging expert-approved scientific modules with runtimes and calibrated numerical checks so that tasks can be generated, executed, graded, and reused across evaluation, supervised fine-tuning, and reinforcement learning.

Key Contributions

1. A construction pipeline that compiles scientific expertise into executable environments. Domain experts define module boundaries, observables, and equivalence contracts; AI agents handle the mechanical work of building, surveying tests, and authoring adapters. A module is defined by scientific responsibility and executable coverage rather than by file layout, and one repository can yield several traceable environments.

2. A calibrated scientific check system decoupled from task authoring. Official tests, regression suites, and shipped examples become checks with explicit pass policies — pointwise bounds of the form |c − r| ≤ a + ρ|r|, or invariants policies for stochastic or rapidly diverging outputs. Each check carries a plain-language warrant explaining which scientific bias the bound distinguishes. Because the check suite is fixed before tasks are written, any task — repair, port, reproduction — is graded by the same contract.

3. A task-factory and validation layer that manufactures verified tasks at scale. Seven authoring categories (Acceleration, Repair, Discovery, Reproduction, Integration, Calibration, Implementation) specialize reusable procedures to each environment. A proposal becomes a task only after executable evidence shows it is observable, solvable, and trustworthy — for injected repairs, a known-valid witness must pass while the defective baseline leaves measurable headroom.

4. A shared episode interface feeding evaluation, SFT, and RL, plus a released model family. The registry holds 2,515 repair, 295 implementation, and 2 acceleration tasks across 64 environments from 27 codebases, with 1,076 companion executable checks. Verified trajectories train PhAI-IDE-72B, PhAI-IDE-9B, and PhAI-IDE-4B.

Main Findings

• Leading agents still fail roughly a third of hard scientific tasks. On the 85-task ScienceIDE-Hard subset (from PLUTO, Athena++, MITgcm, LAPS, and PHANTOM, with codebases of 10⁴–10⁵ lines), Fable 5.1 reaches 67.1%, Opus 5 64.6%, and Astra 63.1%; Sol reaches 55.0%, and the other eleven agents fall below 40%. The authors caution that Fable is singly measured and the top intervals overlap, so the ordering is not statistically established.

• Speed and final accuracy are different axes. Astra reaches 49.6% success at ten minutes versus Fable's 25.9%, but Fable overtakes it around 31 minutes. Between 20 and 60 minutes Astra gains only 2.0 points, while Qwen3.8 Max gains 30.2.

• More spending does not buy correctness. Fable achieves 67.1% at roughly $7.90 per task; Astra achieves 63.1% at $3.56. Astra averages 9.4 minutes and 13.9k output tokens versus Fable's 16.8 minutes and 85.7k tokens. DeepSeek V4.1 Flash emits 172.4k output tokens per task for 36.0% success. Across fifteen agents, Spearman correlations of success with runtime and output volume are −0.22 and 0.01.

• Reference-convention mismatches dominate failures. In trace review of the Fable/Astra runs, failure to match reference conventions accounts for most task-balanced failures — the agent's code runs, but not on the same scientific terms the environment expects.

• Supervised fine-tuning produces real repair gains measured by physics, not by loss. Training on 4,567 verified segments from 564 tasks raises mean repair reward on held-out tasks: 4B on PLUTO-Particles-Dust from 0.0000 to 0.3333; 9B on PLUTO-RMHD/ResRMHD from 0.0000 to 0.2857, on LAPS from 0.3125 to 0.5000, and on MITgcm-Biogeo from 0.0625 to 0.1250.

• Gains transfer beyond science. Fifteen model–benchmark comparisons improved by at least 3 points, spanning code repair (HumanEvalFix JavaScript +10.98 points for 4B), defect detection (CodeXGLUE 0.422 → 0.492), execution prediction, reasoning (BBH Word Sorting 0.240 → 0.576 for 9B; GSM8K), and knowledge (MMLU-Pro CS, ARC-Easy). One confirmation comparison declined, and the authors do not claim uniform improvement.

• Budget truncation must not be scored as failure. In an unmasked RL run, reward initially rose then collapsed below its starting point while tokens per turn fell by more than a factor of three — the policy learned to shorten trajectories rather than solve repairs, and shorter turns increased truncation in a self-reinforcing loop.

• Masking truncated trajectories from the loss while keeping them in the group baseline works. After 30 steps on Qwen3.5-4B, held-out reward on LAPS rose from 0.357 to 0.857 (2.4×) and on MITgcm-biogeo from 0.286 to 0.571 (2.0×). LAPS training reward rose from 0.427 to 0.828 while truncation fell from 39.5% to 6.6%; MITgcm-biogeo went from 0.381 to 0.597 with truncation falling from 34.1% to 23.8% — and mean tokens per turn increased, confirming that the gains came from longer, more productive attempts rather than shorter ones.

Methodology in Plain English

The pipeline has five stages.

Building the environment. An agent inspects a pinned upstream revision of a scientific codebase, resolves dependencies and licences, builds the source, and runs the official tests. Those runs reveal output formats, numerical variability, and expensive paths. The agent then proposes modules — coherent scientific responsibilities with executable coverage — and a domain expert approves the decomposition. An approved module is packaged with an editable workspace for the agent, a set of checks, and a private verifier the agent cannot read.

Turning tests into checks. Every upstream unit test, regression test, and shipped example is traced to a module responsibility and either retained as a check, excluded with a reason, or flagged as a gap. Each retained check gets a pass policy. Pointwise policies compare every graded value against a tolerance; invariants policies compare moments, distributions, conserved quantities, or integral norms when run-to-run variation makes pointwise comparison meaningless. To calibrate tolerances, the authors run nominal and perturbed initial conditions, and where possible an altbuild on a different legitimate build of the same source. Grading targets physical observables (a particle's properties follow its particle ID, not its array slot), never storage order, timings, or random draws.

Specializing factories. Reusable authoring procedures — reversible edits, execution, artifact assembly — are specialized to each environment's active paths, build recipes, and meaningful transformations. Repair and Implementation expose the most automatable candidate expansion via reversible mutation and excision; the other five categories provide interfaces for expert-specified objectives.

Validating tasks. A factory proposal becomes a task only after execution proves it valid. For an injected repair, a known-valid witness must pass, the unfixed baseline must leave headroom, and the change must produce a check failure that the reference repair removes. Ambiguous specifications, unreachable branches, and harness failures are rejected or left ungraded. Repair scores normalize against the defective baseline: r_repair = max(0, (r − f) / (1 − f)), where f is the unfixed build's score. Difficulty is recorded relative to a specific model, scaffold, budget, and date — never as a universal property of the task.

Learning from episodes. Each episode gives the agent a workspace, records its actions and outcomes, and submits artifacts to the private verifier. Evaluation holds tasks and budgets fixed; SFT selects trajectories and supervises agent actions including tool calls; RL connects rollouts to verifier rewards. The RL experiments run generation in vLLM and optimization in PSRL (a customization of the veRL trainer) on disjoint GPUs, with the agent loop executing in shared containers so that rollout continues against current weights while the optimizer runs. The objective is a Dr. GRPO-style group-relative advantage over G = 8 trajectories, without dividing by the group standard deviation, paired with an asymmetric PPO clip (ε_low = 0.2, ε_high = 0.3) so that rarely sampled repair actions can still gain probability mass.

Why This Matters

Impact on research. The paper argues that scientific code is a training substrate, not just an evaluation set. Prior scientific benchmarks — SciCode, ScienceAgentBench, LAB-Bench, DiscoveryBench, CORE-Bench, PaperBench, AInsteinBench, SWE-bench Science — measure agents but do not supply the production infrastructure that manufactures new tasks and verification contracts. ScienceIDE's distinctive move is to fix the acceptance contract per environment and then vary the requested work, so one environment serves repair, implementation, reproduction, and acceleration simultaneously. The transfer results also challenge the assumption that domain specialization costs general capability.

Real-world applications.

• Legacy scientific software modernization. The paper's motivating example is moving scientific codes to accelerators, which currently proceeds one hand-written port at a time, each bound to a vendor toolchain. An agent that can recover validated numerical behavior within tolerance is directly useful for GPU ports of climate, plasma, and astrophysics solvers.
• Reproducibility and code maintenance. Environments built from pinned revisions with calibrated checks can detect when a change silently breaks physics that unit tests never covered — a failure mode the paper notes software tests routinely under-specify.
• Domain-specific scientific copilots. The registry supports filtering by domain, family, or measured difficulty, so a lab can train or select an agent for its own solver and verify it against its own published convergence and conservation criteria.
• Benchmarking and procurement. The outcome–cost profiles (success versus dollars, minutes, and tokens per task) give a more decision-relevant comparison than a single leaderboard number.

Industry relevance. The paper reframes evaluation cost: agents with similar scores differ by more than 2× in dollar cost and nearly 6× in output tokens per task, which matters for anyone running agents at scale. The RL finding about truncation is broadly applicable beyond science — any long-horizon verifier-reward setup where episodes can end by budget cutoff faces the same shortcut, and the fix (mask truncated trajectories from the loss but keep them in the group baseline) is cheap and general. Open infrastructure that lets models and trainers change without rebuilding scientific content lowers the barrier for organizations that want to fine-tune on proprietary simulation code without publishing it.

Future Directions

• Expand beyond repair and implementation. Repair (2,515) and implementation (295) tasks dominate the registry, while acceleration has only two tasks and Discovery, Reproduction, Integration, and Calibration remain interfaces rather than populated generators. Scaling those categories requires expert-specified objectives that resist mechanical expansion.

• Diagnose and reduce reference-convention failures. Convention mismatches dominate the failure analysis; the open question is whether these are addressable through better task specifications, richer verification feedback during episodes, or training data that exposes the conventions explicitly.

• Strengthen the credibility of the transfer claim. The public-benchmark results are suggestive but not uniform, include at least one decline, and come from 15 screening comparisons plus a smaller confirmation set on disjoint items. Larger confirmation studies and a mechanistic account of why scientific trajectories help general reasoning would materially strengthen the case.

• Scale and lengthen the RL evidence. The RL study covers two environments, one base checkpoint, and 30 steps. Whether the truncation-masking fix and the observed 2× held-out gains hold across more simulators, longer training, and larger models is untested.

• Broaden the evaluation set. ScienceIDE-Hard is 85 tasks from five physics- and geoscience-oriented codebases. Coverage of chemistry, biology, and materials science is a stated goal but not yet demonstrated at comparable depth.

Target Audience

Most valuable for: researchers and engineers building agent training infrastructure, especially those working on verifiable-reward RL, SWE-style repository agents, or scientific computing applications. Also directly relevant to ML platform teams deciding how to turn internal codebases into training and evaluation data.

Also useful for: computational scientists and research software maintainers interested in automated verification of numerical correctness; benchmark designers concerned about expert-authoring cost, saturation, and contamination; and technical decision-makers comparing agent deployments on cost-per-task rather than accuracy alone.

Prerequisites: comfort with standard ML training concepts (fine-tuning, LoRA, RLHF-style objectives) for the learning sections; no specialized physics background is required, though the check-calibration discussion assumes familiarity with numerical tolerance and reproducibility concerns.

Authors’ abstract

Scientific code repositories encode decades of human knowledge in executable models, methods, and tools. Yet fragmented toolchains, implicit domain conventions, and specialized correctness criteria make this knowledge difficult to convert into reliable learning experience-a challenge we call the scientific experience bottleneck. We introduce ScienceIDE, infrastructure for turning the world's scientific code into programmable environments for scientific agents. Guided by expert-defined scientific cases and acceptance criteria, agents transform repositories into executable environments that support task generation, execution, and scientific verification. These environments provide a shared foundation for supervised fine-tuning, reinforcement learning, and evaluation. Using verified interaction trajectories, we train PhAI-IDE-72B, PhAI-IDE-9B, and PhAI-IDE-4B. The model family shows gains in held-out scientific-code repair and across selected general-purpose benchmarks in code, reasoning, and knowledge, providing evidence of positive transfer from scientific experience to broader capabilities. ScienceIDE lays the foundation for an integrated workspace for agent learning and scientific practice, making humanity's scientific software a shared substrate for developing scientific intelligence. Code: https://github.com/aitofound/ScienceIDE