Self-Supervised Scaling of Terminal Environments for Scientific Domains

AI summary

Overview

Research area: Self-supervised training-data construction for LLM terminal agents in scientific computing (cs.SE, with ties to agent training and program synthesis).

Technical level: Intermediate, moving to Advanced in the verification and benchmark-design sections. Readers need familiarity with supervised fine-tuning, terminal-agent scaffolds, and the notion of a test oracle, but the core idea is explained through examples.

Scope: The paper introduces "software-in-the-loop reconstruction," a method that turns existing scientific command-line programs into training environments and verified trajectories for terminal agents, instantiated as a 3,000-task, six-domain benchmark called SWR plus a fine-tuning corpus.

What This Paper Is About

Terminal agents are increasingly used outside software engineering, but training and evaluating them in science requires two hard-to-build things per task: a reference solution and a domain-specific checker that can tell a genuinely correct numerical, geometric, or tabular output from one that merely looks valid. The authors' goal is to obtain both from a single source so that new tasks can be added without hand-authoring a solution and a validator for each one.

Their answer is to treat an existing scientific program (a "workflow") as the ground truth: run it on several input configurations, release some input–output pairs to the agent, hide the rest, and ask the agent to write a program that reproduces the workflow's behavior on the hidden configurations. Programs that pass become supervised fine-tuning data.

Key Contributions

1. The software-in-the-loop reconstruction framework. The paper formulates task construction so that one executable workflow supplies the reference solution, the public examples, and the hidden reference outputs for multiple task variants, instead of specifying the solution and the verification target independently for every task.

2. The SWR dataset and a hierarchical semantic verifier. SWR contains 3,000 tasks built from 500 workflows and 46 software families across six domains (including physical and engineering simulation, life and molecular sciences, and earth and space sciences, plus formal, spatial, and data workflows). The verifier combines domain-specific semantic comparison, structural validity, and anti-shortcut checks.

3. Empirical validation of the construction. The authors measure how strongly workflow outputs respond to input changes, test predefined public-output replay controls, and run an independent audit of the frozen verifier, including a blinded human review.

4. Evidence that verified trajectories are useful training data. Reconstruction-only supervised fine-tuning of Qwen3.8-27B is evaluated over three seeds and against four matched-token corpus controls, with gains on Terminal-Bench 2.0 (TB2), Terminal-Bench 4.0 (TB4), Long-Horizon Terminal-Bench (LHTB), and a held-out 100-task subset called SWR100.

Main Findings

• Scheduled release size and composition: 3,000 tasks, 46 software families, 500 workflows, 15,600 public scenarios, and 8,400 hidden scenarios; no family contributes more than 2.4% of tasks. The release averages 5.2 public scenarios and 2.8 hidden scenarios per task.

• Workflow outputs respond to input changes: every compared configuration pair changes the output for 2,984 tasks, and for the remaining 16 tasks, 35 of the 36 pairs change. Across all tasks, 75.1% of the output fields used to assess behavior vary with the input configuration, and an average pair differs in 66.3% of those fields. On 2,400 tasks, varying one declared parameter while holding the others fixed changes the output for 89.5% of declared parameters.

• Replay shortcuts do not pass: the three public-output replay controls (modal replay, nearest replay, and an oracle best-public replay that consults the hidden reference) reach 34.3–44.3% agreement on hidden output fields but satisfy the strict acceptance requirements on no released task. The authors state explicitly that this zero-success result restates the release selection rule and does not demonstrate resistance to shortcuts beyond those three tested.

• Verifier agreement with humans: the frozen verifier accepts 2 invalid cases and rejects 2 valid cases out of a blinded 200-case review (100 valid, 100 invalid), corresponding to a 2.0% false-acceptance rate and a 2.0% false-rejection rate. Development used 35,873 controlled cases to set tolerances before freezing.

• Single-attempt agent performance is low: Qwen3.8-Max (with the Terminus-2 scaffold, at most 100 turns and one hour per task) solves 685 tasks, a 22.8% Pass@1 over all 3,000 scheduled tasks; records exist for 2,978 tasks (99.3% record rate), on which the rate is 23.0%.

• Failure is behavioral, not just executability: of 2,781 evaluated workspaces containing an executable candidate, 1,591 emit the same output across different hidden scenarios and 75 replay a public output. Successful runs end at a median of 58 turns versus 85 turns for failures attributed to the model, which the authors describe as an association rather than evidence that length causes failure.

• A large gap remains for current models: across eight API-served models on the 100 SWR100 tasks (at most 500 turns and three hours per task), GPT-5.6 Sol reaches 20.0% Pass@1 and the other models fall between 2.0% and 6.0%. Of 44 successful runs, 43 fall in Software & Formal Systems, Physical & Engineering Simulation, or Spatial, Visual & 3D; only one success occurs in Life & Molecular Sciences, and none in Data & Document Workflows or Earth & Space Sciences.

• Repeated sampling yields a training corpus: across three attempts per task, Qwen3.8-Max solves 838 tasks at least once (27.9% Pass@3), producing 1,422 verified trajectories, which are oversampled into a 3,000-entry reconstruction-only training corpus.

• Fine-tuning improves terminal-agent benchmarks: at the 3,000-entry budget, Qwen3.8-27B improves from 47.94% to 53.56% on TB2, from 0.51% to 3.54% on TB4, from 20.67% to 27.67% on LHTB, and from 1.33% to 3.33% on SWR100. Mean performance is non-decreasing across the 750, 1,500, and 3,000 entry budgets.

• Reconstruction data beats matched-token alternatives: at an identical budget of 88.48M assistant-loss tokens, reconstruction SFT attains the highest mean on all four evaluations among the five corpora compared (OpenThoughts-Agent-v1, TMax, LiteCoder-Terminal, Nemotron-Terminal, and reconstruction). For example, TB2 means are 53.56 for reconstruction versus 49.06, 49.44, 48.69, and 47.19 for the four controls.

• Qualitative failure case: in an OpenSCAD snap-fit task whose public example is 16 mm and hidden reference is 21 mm, GPT-5.6 Sol is the only one of eight models whose output preserves separate body and lid components; the other seven collapse to plate-like meshes at wall-thickness height, producing valid geometry that does not reproduce the workflow's dependence on the input.

Methodology in Plain English

The authors start from programs that already exist in scientific practice: simulators and analysis pipelines that take structured inputs and produce outputs. Each such program is adapted once to a common interface, so it can be reused as many tasks rather than one.

For each workflow they generate typed input configurations (numeric, categorical, or path-valued fields) and split them into two disjoint sets. The public set is released to the agent as input–output examples, along with an instruction and the input schema. The hidden set is kept private and is executed only to produce reference outputs for scoring. Each workflow yields six task variants that differ in which examples are released and which are hidden.

An agent then works in a terminal to write its own editable program that reproduces the workflow's behavior, without seeing the workflow source. Public tests give feedback during revision. A hierarchical judge scores the candidate against the workflow's outputs, and acceptance requires a perfect hidden score plus structural validity and anti-shortcut checks that reject replaying public outputs or ignoring the inputs.

A task is admitted to the release only after several gates: the workflow must execute successfully, be deterministic, have a stable output schema, and respond measurably to input changes; the wrapped workflow itself must pass; and all three replay controls must fail.

Trajectories whose final program meets the strict criterion are collected for supervised fine-tuning, with any task identifiers appearing in an evaluation set removed first. Because training entries are sampled with repetition, the corpus is a multiset in which a repeated trajectory is reweighted rather than treated as an independent solution. The fine-tuning objective is standard next-action prediction over recorded interaction histories. Comparisons are run at a matched assistant-loss-token budget so that gains cannot be attributed to more compute.

Why This Matters

The work reframes where verified supervision comes from. Rather than writing a checker for every task, it lets an existing scientific program be both the answer key and the source of hidden test cases, which is precisely the bottleneck for extending agent training beyond software engineering into domains where "correct" is a numerical tolerance, a geometric equivalence, or a domain-specific invariant.

Real-world applications include:

• Scientific simulation and engineering analysis: tasks built from CAD and geometry tools (for example OpenSCAD), robot description repair (URDF mass and inertia fields), and simulation pipelines that produce meshes and arrays.
• Molecular and life sciences tooling: workflow reconstruction over bioinformatics and molecular analysis programs, a domain where the paper finds current agents nearly unable to succeed.
• Earth and space science data processing: the domain where no evaluated model achieved a strict success on SWR100, marking a concrete capability gap.
• Agent-training data pipelines: organizations that need domain-specific executable environments can wrap their own internal programs and inherit the same release checks, since the method adds workflows rather than redesigning tasks.

Industry relevance is direct: the construction scales by adding workflows and input configurations under fixed checks, and the resulting fine-tune improves general terminal-agent benchmarks (TB2, TB4, LHTB) without mixing in auxiliary terminal-task data and without sharing a training source with those benchmarks.

Future Directions

• Generalization to unseen workflows. SWR100 measures held-out instances of the same reconstruction setting, and the authors explicitly note it does not measure generalization to workflows never used to build tasks. Testing transfer to genuinely new scientific software is the open question.
• Separating unique trajectories from repeated sampling. Because the corpus is oversampled and each repeated copy enters the loss separately, future work needs to disentangle how much of the gain comes from additional distinct solutions versus reweighting a small pool.
• Broadening anti-shortcut coverage. The paper acknowledges that its zero-success replay result covers only the three controls tested, leaving room for shortcut families that were not enumerated, and for broader verifier stress testing beyond the 35,873 development cases and the 200-case blinded review.
• Beyond terminal-skill transfer. The downstream evaluations measure terminal-agent transfer rather than autonomous scientific discovery, so connecting reconstruction ability to actual scientific task completion remains unaddressed. The authors also flag that agreement on a finite held-out set is not recovery of the original implementation or the underlying mechanism.

Target Audience

Researchers and engineers building agent training data or executable benchmarks, especially those working on terminal agents, code agents, or domain-specific evaluation in science and engineering. It is also relevant to practitioners who maintain scientific software and want to repurpose existing pipelines as evaluation or supervision infrastructure, and to readers interested in test-oracle problems, programming by example, and behavioral system identification. Readers focused on pure benchmark leaderboards will find the methodological and verification-analysis sections most useful; readers focused on model training will find the SFT scaling and matched-token control results most useful.

Authors’ abstract

Terminal agents are increasingly deployed beyond software engineering in science and other specialized domains. Constructing training environments requires executable reference behavior and a domain-specific verifier that distinguishes semantic correctness from superficially plausible artifacts. Authoring these components for each task requires repeated engineering and limits reuse. We introduce software-in-the-loop reconstruction, a self-supervised framework that obtains reference outputs and verification targets from existing software workflows, executable programs mapping structured inputs to outputs. For each workflow, we execute multiple input configurations and partition cases into public observations and hidden evaluations. Given the instruction, input schema, and public input--output observations, an agent constructs an editable program without access to the source workflow. The candidate is evaluated on hidden configurations against workflow outputs. A hierarchical verifier combines domain-specific semantic comparison, structural validity, and anti-shortcut checks, while public feedback supports iterative revision. The construction admits additional workflows and configurations without authoring a reference solution for each task. We instantiate SWR with 500 workflows and 46 software families across six domains. Across three attempts per task, Qwen3.8-Max solves 838 tasks and produces 1,422 verified trajectories, which we oversample to 3,000 reconstruction-only training examples. Supervised fine-tuning of Qwen3.8-27B improves mean Terminal-Bench 2 performance from 47.94% to 53.56% across three seeds and achieves the highest mean among four matched-token corpus controls on all four reported evaluations. These results indicate that existing scientific software can provide scalable, behaviorally verified supervision for terminal agents.