CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

AI summary

Overview

Research area: Machine learning — autonomous synthesis of executable training data and fine-tuning of LLM terminal agents.

Technical level: Advanced.

Scope: This paper introduces CalibForge, a system that builds 5,431 verifiable terminal-agent tasks by using the observed pass/fail behavior of solver models to adversarially revise candidate tasks before they are used for supervised fine-tuning.

What This Paper Is About

Existing systems can automatically construct terminal tasks and check that they are executable and solvable, but executability says nothing about whether a task is appropriately hard for a given model — it may be trivial or effectively impossible. This paper treats task learnability as solver-relative: a task is worth training on when at least one solver succeeds and at least one fails. CalibForge implements this as a constrained adversarial loop in which an authoring agent revises tasks (instructions, environments, or verification tests) based on verified solver outcomes until a target pass/fail pattern is reached, then distills trajectories from the retained tasks to fine-tune Qwen backbones.

Key Contributions

1. Environment-level behavioral calibration — a construction principle that makes task learnability solver-relative, using verified solver behavior both as the retention signal and as the driver of multi-round revision.
2. CalibForge, an autonomous terminal-task synthesis system that realizes the principle through a constrained adversarial author–solver loop, with two calibration strategies: multi-solver calibration (targets cross-solver disagreement) and contrastive solver calibration (targets a designated strong-pass/weak-fail relation).
3. A collection of 5,431 calibrated terminal tasks (1,263 from multi-solver calibration and 4,168 from contrastive calibration) spanning all 16 categories in the authors' domain taxonomy.
4. Evidence that calibration beats authoring-and-validation alone or single-solver feedback, on Terminal-Bench 2.0 and on the out-of-distribution benchmarks SWE-bench Pro and Doc2Repo.

Main Findings

• Calibrated data improves terminal-task performance. CalibForge-30B-A3B and CalibForge-35B-A3B reach 32.58% and 47.57% on Terminal-Bench 2.0, surpassing the strongest baselines under a shared training protocol by 6.36 and 6.75 percentage points.
• Largest gains occur over the corresponding base models. Improvements reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo.
• Gains transfer out of distribution. The 35B model achieves 44.32% Resolved Rate on SWE-bench Pro (731-task public set) and 48.77% Pass Rate on Doc2Repo, the highest among the evaluated training-data sources for the same backbone. CalibForge-30B-A3B improves over its base model by 27.68 points on SWE-bench Pro and 30.04 points on Doc2Repo; CalibForge-35B-A3B improves by 3.03 and 3.85 points respectively.
• Calibration matters more than solver feedback or trajectory volume. In a matched 1,300-task ablation on Qwen3-30B-A3B-Instruct, single-solver feedback yields 24.34% on Terminal-Bench 2.0 (+1.87 over the 22.47% no-solver variant), while multi-solver calibration reaches 29.21% (+6.74) and contrastive calibration reaches 31.09% (+8.62). Multi-solver calibration produces fewer retained SFT trajectories (2,425) than the no-solver variant (2,466) yet improves accuracy substantially.
• Validity checks do not place tasks in the learnable zone. Of candidates entering contrastive calibration — all of which had passed structural validation and self-solving — only 19% initially satisfied the target strong-pass/weak-fail relation; after feedback-driven revision and re-probing, 96% ultimately satisfied it. The dominant initial pattern was both-solvers-pass, indicating insufficient separation between solver settings rather than unsolvability.
• Calibration corrects mismatches rather than filtering tasks. Grouping runs by recorded solver probes, 15% of retained runs complete with one probe, 53% within five probes, and 93% within twenty — an early-correctable majority plus a long tail requiring sustained probing.
• The task collection is broad in several dimensions. The largest domain category is software engineering at 25.5%, compared with system administration at 74.6% of SETA-Env and 49.9% of TerminalTraj, debugging at 67.0% of CLI-Gym, and file operations at 40.6% of Endless-Terminals. The collection contains 3,885 distinct capability tags with a median of five per task; 51.6% of tags occur in only one task and 82.2% in at most five. A median task has two initial artifacts, one distinct file type, two environment dependencies, and seven verifier test functions, with 90th-percentile values of eight, four, seven, and 15.
• Interaction length and reasoning depth are not the same thing. Under an identical teacher protocol, CLI-Gym has the highest median interaction length at 28 steps versus 21 for CalibForge, but CalibForge has the highest median teacher thinking-token count at 5.3k versus 4.0k for CLI-Gym.
• Per-category gains are broad on Terminal-Bench 2.0. Both CalibForge variants improve or match their base backbones in every category (pass@3 comparison in the paper's Figure 3).

Methodology in Plain English

CalibForge starts from a short clue (a domain, scope, and capability and tool hints) and hands it to an authoring agent powered by DeepSeek-V4-Pro, working in a draft sandbox with web search, shell, and file-editing tools. The agent researches official documentation, GitHub repositories and issues, and Stack Overflow for concrete engineering problems, picks a direction on relevance, distinctiveness, and feasibility, tests its assumptions in the sandbox, and then writes the task instruction, the execution environment (Dockerfile and initial files), and the verification tests.

Every candidate then passes a two-stage gate. Structural validation builds the environment, runs the verifier, and confirms all tests fail in the initial state. Self-solving has the authoring agent attempt the task in an isolated sandbox and confirm the intended solution works and that instruction, environment, and tests agree. Candidates that fail either stage are repaired and re-checked.

Only then does adversarial calibration begin. In each round, CalibForge spins up an isolated sandbox per solver, gives every solver the same instruction, runs the verifier against the final sandbox state, and records a verified outcome (all tests pass or not), along with a structured feedback summary (outcome, step count, completion status, self-assessment, failure diagnosis) and the full interaction trajectory.

• Multi-solver calibration dispatches K solvers using different models. The candidate is retained only if at least one passes and at least one fails.
• Contrastive solver calibration runs a designated stronger and a designated weaker solver, retaining the candidate only when the stronger passes and the weaker fails.

If the criterion is unmet, the authoring agent uses the feedback to revise any part of the task — including returning to web research — and the candidate is revalidated and re-probed, for at most R_max rounds before being discarded.

Experiments use DeepSeek-V4-Pro as the author, DeepSeek-V4-Flash, GLM-5, and Kimi K2.5 as the three multi-solver subagents, and DeepSeek-V4-Pro / DeepSeek-V4-Flash as the stronger/weaker pair. Each solver attempt is capped at 100 interaction steps and 30 minutes; R_max is 50.

Retained tasks are stored as Harbor-style instances and trajectories are distilled with DeepSeek-V4-Pro at high reasoning effort using CalibForge-Eval, a minimal scaffold exposing only bash, file-editing, and finish tools. Each task is attempted twice with a 200-step limit and one-hour timeout; passing trajectories are filtered for length, invalid tool calls, and tokenizer-unsafe tokens. Qwen3-30B-A3B-Instruct and Qwen3.5-35B-A3B are fine-tuned with full-parameter SFT for 10 epochs. Baselines (Endless Terminals, SETA-Env, CLI-Gym, TermiGen, TerminalTraj) are re-distilled from their released task sets under the identical teacher protocol and recipe. Evaluation uses CalibForge-Eval for Terminal-Bench 2.0 with a 500-step limit, one-hour timeout, and 16 CPU / 32 GB RAM sandboxes; SWE-bench Pro and Doc2Repo use their official scaffolds. Training tasks are decontaminated against all three benchmarks using 14-gram prompt overlap plus 5-shingle Jaccard similarity over normalized instructions and verifier code, combined with shared output paths, overlapping test functions, and high-risk task-family matches.

Why This Matters

Impact on research. The paper shifts the target of data synthesis from "is this task valid?" to "where does this task sit relative to a given solver setting?" That reframing turns solver disagreement into a construction-time signal for supervision, and the ablations separate the two calibration strategies from generic solver feedback and from trajectory volume. It also suggests that dataset size and domain breadth alone are insufficient descriptors of a training collection, since capability diversity and environment/verification structure capture complementary properties.

Real-world applications:

• Repository-level software engineering: issue resolution in existing repositories and full-repository generation from natural-language specifications, the two out-of-distribution benchmarks used here.
• System administration and terminal operations: multi-step shell workflows, configuration and dependency diagnosis.
• Data processing and recovery: the paper's worked example recovers environmental readings from a corrupted binary sensor log via record synchronization and CRC-16 validation, discarding records with invalid checksums or impossible values.
• Security engineering: another worked case repairs cryptographic vulnerabilities in a legacy command-line password manager while keeping existing vault entries readable.

Industry relevance. Teams building coding and terminal agents need large volumes of executable, verifiable, appropriately difficult training tasks; CalibForge offers an automated pipeline for producing them and a criterion for deciding which candidates to keep, rather than relying on human-authored difficulty judgment.

Future Directions

• Calibration budget as a selection effect. The paper notes that the maximum calibration horizon determines not only construction cost but which recoverable candidates enter the training set; a short horizon favors candidates whose mismatches are correctable in few revisions. How to choose that budget, and how it changes the resulting data distribution, remains open.
• Handling the long tail of mismatches. Only 15% of retained contrastive runs complete within one probe and 93% within twenty; what to do with the candidates that need sustained calibration, and whether they are worth the compute, is unresolved.
• Generalizing solver-relative learnability beyond terminal tasks. The principle is defined for solvers in terminal sandboxes; whether the same retention criteria transfer to other agentic domains with verifiable outcomes is not established.
• Choosing and composing the solver pool. The paper uses a three-model heterogeneous pool for multi-solver calibration and one designated strong/weak pair for contrastive calibration; how the composition of that pool, or the size of the strong–weak gap, shapes the difficulty of the retained tasks is not systematically studied.

Target Audience

Researchers and engineers working on LLM agents, agentic training-data synthesis, and executable benchmark construction; practitioners who fine-tune coding or terminal agents and need automated pipelines for verifiable, appropriately challenging task generation; and evaluation researchers interested in how solver behavior can be used as a construction signal rather than only as a final metric.

Authors’ abstract

Training terminal agents requires executable and verifiable tasks that are not merely solvable, but appropriately challenging for learning. Executable validation establishes feasibility, yet does not reveal how a task behaves relative to a given solver setting. In this paper, we present CalibForge, an autonomous terminal-task synthesis system that uses verified solver behavior to revise candidate tasks through adversarial solver calibration. Multi-solver calibration targets disagreement within a heterogeneous solver pool, whereas contrastive solver calibration targets a designated strong-pass/weak-fail relation; both operationalize a solver-relative learnable zone anchored in demonstrated solvability. Using CalibForge, we construct 5,431 calibrated terminal tasks. Our ablations show that both strategies yield more effective supervision than authoring and validation alone or ordinary single-solver feedback. Models trained on the full collection achieve 32.58% and 47.57% on Terminal-Bench 2.0. The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. Together, these results support solver-relative learnability as a practical target for constructing effective and transferable agent training data.