Rules to Tools: Executable Checks for LLM Agents in Scientific Computing

AI summary

Overview

Research area: AI coding agents for scientific computing, specifically how the form in which a scientific verification rule is delivered (written text versus a ready-to-run executable checker) affects an agent's ability to repair faulty simulation and numerical code.

Technical level: Advanced. The paper assumes familiarity with LLM agent repair loops, benchmark evaluation protocols, PDE solvers, and quantum-chemistry-style numerical routines (Numerov recurrence, spherical harmonics, density-response integrals), though the core comparison is easy to grasp.

Scope in one sentence: A controlled, matched-group study that asks whether packaging public scientific requirements as callable executable checks improves LLM-agent repair of scientific programs, at what token and CPU cost, and on which tasks.

What This Paper Is About

Scientific coding agents receive equations, boundary conditions, and output requirements as text, and must judge whether the program they just revised actually satisfies those requirements. The authors introduce Rules to Tools (R2T), which converts written public scientific requirements into prepared, callable checking tools that an agent can invoke on its current program at any point during revision. The paper's central question is whether supplying that ready-to-run implementation — while holding the written rule, starting program, model, and budgets fixed — changes repair outcomes and resource cost.

Key Contributions

1. A packaging method. R2T turns public scientific requirements into author-constructed, executable diagnostics that measure the program under revision. Each diagnostic is specified as a tuple of (relation, probe, observation, status), covering boundary values, physical relations, equations, geometry, and required output files.

2. A matched SciCode repair design. The study holds written criteria, starting programs, model, and budgets fixed while varying only access to a prepared implementation, with an independent evaluator grading final complete modules. Additional arms vary how the check is delivered (an initial report, or the checker source run through ordinary Python).

3. Measurement across several benchmarks. The paper reports complete repair, native-step (partial) progress, and resource cost for matched SciCode comparisons plus complementary studies in PDE, molecular-simulation (MDArena), and repository (AInstein) settings.

4. Checker qualification evidence. For the seven-task extension, 21 draft checks were generated, 14 were selected; they passed 28/28 accepted-program evaluations, 8/8 additional evaluations on programs kept outside selection, and detected 7/7 targeted seeded faults.

Main Findings

• Small aggregate gain on matched SciCode task IDs. Across the two task-ID cohorts, complete repair is 26/30 with text and 29/30 with prepared checks. The eight-ID cohort scores 13/16 versus 15/16; the seven-ID extension scores 13/14 versus 14/14.

• The gain is task-dependent and driven by a few tasks. In the eight-ID cohort, two tasks favor tools, one favors text, and five tie. Tasks 17, 77, and 11 favor tools; task 37 favors text; eleven tasks tie overall. Task 77 goes from 0/2 with text to 2/2 with tools; task 77 alone accounts for the whole net effect on that cohort.

• Uncertainty is wide. The task-macro tool-minus-text difference in the eight-ID cohort is 12.5 percentage points, with a task-cluster bootstrap 95% interval of [-12.5, 43.75] percentage points (10,000 draws, seed 20260923). Removing task 77 gives zero difference across the other seven tasks, with interval [-21.43, 21.43]. The seven-ID extension has interval [0, 21.43]. Exact two-sided sign tests give p = 1 for the eight-ID (2:1) and seven-ID (1:0) cohorts, and p = 0.625 for their combined 3:1 pattern.

• Gains occur with and without an initially flagged error. Initial reports flagged tasks 17 and 48. Task 77's initial checks reported no violation, as did task 11's two initial checks in the extension; both successful task 77 tool trajectories later repaired periodic wrapping and pressure through ordinary Python. Task 17's gain occurs with a flagged initial violation. The authors describe this as measuring the full executable-support package, not just re-flagging known faults.

• Delivery form matters less than access in one arm. A fresh source-through-Python arm reaches 15/16, matching the dedicated-command arm, while the initial-report arm reaches 14/16 and written rules alone 13/16. The source-through-Python arm produced 40 completed public checker reports.

• Larger and alternate-program cohorts. The twelve-task shared-definition SciCode cohort ties at 13/24 per group. Five development-exposed tasks with alternate starting programs score 3/10 with text versus 7/10 with checks, and task-macro native subtask completion rises from 61.27% to 90.48%. Task 80 stays incomplete in both groups, but text trajectories pass 6/7 and 5/7 subtasks while both tool trajectories pass 6/7.

• PDE results. In a matched PDE comparison, detailed text scores 23/24 and checks score 24/24, with 31.2% lower reported model output for checks (1,062,099 versus 730,723 output tokens; 25.0% bound). Brief-guidance PDE versus rules plus checks ties at 24/24 in both.

• Flow comparison changes more than one thing. Under the primary 24-response cap, flow completes 16/28 text and 21/28 tool trajectories (signs 5:1:1); the 25-response sensitivity gives 20/28 and 24/28 (signs 4:2:1). The scalar-PDE pilot completes 4/4 in both groups. The authors note this selected cohort also changes rule-card delivery and the field-capture gate, so it is not a clean single-variable contrast. Selection was on text-probe failure, which the authors state favors the tool group.

• Historical panels. MDArena scores 5/18 versus 7/18 and AInstein 4/15 versus 7/15 for written rules versus rules plus checks.

• Costs move in opposite directions. In the eight-ID cohort, text-to-tool totals are 7.25M to 7.88M model tokens (+8.7%), 767,871 to 819,981 output tokens (+6.8%), and 456.90 to 633.15 public CPU seconds (+38.6%). The seven-ID cohort uses 5.78M to 4.52M total tokens and 322.53 to 826.73 CPU seconds. Across both cohorts, reported output is at least 1,327,295 text tokens versus 1,262,059 tool tokens (at least 4.9% lower for tools), while public CPU rises from 779.43 to 1,459.88 seconds (+87.3%). One seven-ID text request has no returned usage.

• Output savings in other cohorts. Development-exposed SciCode saves 22.9% reported output (17.9% bound). Under the primary 24-response flow cap, text uses 1,649,905 output tokens and tools use 1,786,229, with every request recorded.

Methodology in Plain English

The researchers define a scientific requirement as something measurable: a relation (the rule), a probe (inputs to test it with), an observation (the discrepancy or error), and a status (a verdict, a continuous number, or "unavailable"). They then write both a written description and an executable implementation of the same checking content, which they call specification equivalence at the level of declared criteria and probes.

They run matched experiments. Two groups of agent runs get the same starting program, the same written rules, the same model, the same budget, and the same access to ordinary Python. The only difference is that the tool group can additionally call a prepared checker through a dedicated command. Independent graders, which never see the interaction, score the final complete module.

Budget and execution settings are fixed: DeepSeek Flash, three saved revisions, 196,608 reported output tokens, and 360 CPU-budget seconds per trajectory, with a 120-second wall cap and 3 GB memory cap per public execution, and a primary response cap of twenty-four. SciCode grading requires every scored subtask to pass on the 2024 SciCode-Verified tests, retrying only failed steps in the pinned 2025 environment; a step passes if either environment accepts it. PDE requires relative grid error at most 10⁻³ and solve-call time at most 30 seconds. Partial credit uses task-macro averaging.

Checks were frozen before paired repair, and the paper reports qualification: static preflight, execution on starting modules, accepted-program validation, and deliberately seeded faults. The authors state that checker authoring time was not recorded and that no blinded authoring log was retained.

Why This Matters

Impact on research. The paper shifts attention from "does the agent write correct code?" to "does the agent have a repeatable way to measure the program against public scientific requirements?" It provides evidence that the delivery form of a rule — text versus a callable check — can change outcomes on specific tasks, and it quantifies the token and CPU consequences of that choice rather than treating verification as free.

Real-world applications:

• Simulation code in physics and engineering: agents that revise PDE solvers and must respect boundary conditions and equation residuals between edits.
• Molecular simulation workflows: MDArena-style tasks where structure, protocol, and produced artifacts must satisfy domain constraints.
• Scientific software repositories: AInstein-style tasks where public APIs and domain probes define correctness for agent-authored changes.
• Calculated observables in materials and quantum workflows: checks such as radial recurrence residuals, field reconstruction after rotation, and density-response integrals, which the paper implements as reference computations.

Industry relevance. Agent-side token savings (4.9% lower reported output across the two SciCode cohorts, 22.9% in the development-exposed cohort, 31.2% in matched PDE) matter for cost, but the paper shows public CPU use rising (+87.3% pooled), meaning the savings shift cost rather than eliminate it. Teams building agent harnesses for numerical and scientific work can use this tradeoff to decide when a prepared checker is worth shipping.

Future Directions

1. Measure checker authoring cost. The paper states explicitly that checker authoring time was not recorded and that no blinded authoring log was retained. Quantifying the human effort to build checks would clarify when R2T is economical.

2. Disentangle delivery form from capture gate. The flow comparison changes rule-card delivery, implementation access, and the field-capture gate at once, and selection was on text-probe failure. A design that varies only one of these would isolate the cause of the 5:1:1 and 4:2:1 task signs.

3. Determine when a task benefits and why. The eight-ID effect collapses to zero without task 77, and two of the three tool-favoring tasks (77 and 11) had no initially flagged violation. Identifying the task properties that predict benefit — beyond "the checker exposes the right measurement" — remains open.

4. Reduce the CPU overhead. Public CPU rose from 779.43 to 1,459.88 seconds pooled across the two SciCode cohorts (+87.3%) while output tokens fell. Cheaper checker execution, caching, or selective invocation would make the support package more attractive at scale.

Target Audience

Researchers and practitioners building or evaluating LLM agents for scientific and numerical code: agent-harness engineers, benchmark designers for scientific coding (SciCode, PDE, molecular, and repository settings), and computational scientists interested in whether executable verification can be packaged as a reusable tool rather than re-implemented as text instructions. Readers focused on broadly generalizable agent improvements should note the wide bootstrap intervals and the task-cluster sign tests reported here.

Authors’ abstract

Scientific coding agents receive equations, boundary conditions, and output requirements in writing, then must assess the programs they revise. Rules to Tools (R2T) supplies prepared executable checks of public scientific requirements. Matched SciCode repair groups share written checks, starting programs, model, and budgets; the tool group receives a callable implementation. Across two task-ID cohorts, complete repair is 26/30 with text and 29/30 with the prepared checks. Three task IDs favor tools, one favors text, and eleven tie. The eight-ID cohort scores 13/16 versus 15/16, with a task-cluster bootstrap 95% interval of [-12.5, 43.75] percentage points for the difference. The larger shared-definition SciCode cohort ties at 13/24 per group. Five development-exposed tasks with alternate starting programs score 3/10 versus 7/10. The tool group favors tasks 17, 77, and 11; initial checks flag task 17 and report no violation for tasks 77 and 11. Task 37 favors text and has no initial reported violation. A fresh source-through-Python arm also reaches 15/16, matching the dedicated command's aggregate. In a matched PDE comparison, detailed text scores 23/24 and checks score 24/24, with 31.2% lower reported model output for checks. Agent-side output savings vary by cohort, while public CPU use rises in both task-ID cohorts. These results measure task-dependent repair outcomes and agent-side costs with prepared checks.