AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?

AI summary

Overview

Research area: Automated synthesis of training data for language-model agents, and the measurement of that capability within a recursive self-improvement (RSI) loop.

Technical level: Advanced. The paper assumes familiarity with supervised finetuning, reinforcement learning from verifiable rewards, agentic task environments, pass-rate-based difficulty calibration, and benchmark construction.

Scope in one sentence: AutoDataBench turns the acceptance decision a data pipeline applies to a single synthesized agentic task into a scored evaluation episode, and measures whether frontier agents can author usable training tasks without any training run.

What This Paper Is About

Progress in language models has come more from data than architecture, but producing the verifiable agentic tasks that feed supervised finetuning and reinforcement learning still depends on human experts. Automating that production would let data scale with compute rather than with expert headcount, and would supply a step of recursive self-improvement. Existing evaluations of "agents writing data" mostly measure a model's performance after training on what the agent produced — which does not match how the data industry works, where samples are delivered and accepted one at a time against criteria, before any training. AutoDataBench asks the missing question: does one individual synthesized task meet a production pipeline's acceptance standards on validity, novelty, difficulty and behavioural coverage?

Key Contributions

1. A per-artifact evaluation target for data synthesis. AutoDataBench scores one delivered executable task at a time, judged before any training run, reproducing the delivery granularity of a real data team instead of the batch-level score of a training run.
2. A three-term episode score tied to acceptance criteria. Score = gate x difficulty x quality, where the gate rejects eight disqualifying defects, difficulty is set by the target model's measured pass rate on the delivered task, and quality is coverage of a hidden rubric of behavioural "modes" the author agent never sees.
3. An instantiated benchmark across three suites. Twenty-four original tasks (eight each) spanning terminal work, software engineering, scientific computing and business workflow automation, drawn from Terminal-Bench 4.0, Terminal-Bench-Science and AutomationBench, with 144 sanitised target-model attempts (K = 6 per task) as the shared evidence base.
4. Evidence that current agents can write the tasks but cannot calibrate them. No evaluated agent scores above 20 out of 100 at the default 45-minute budget; difficulty is the binding constraint while rubric coverage is near-saturated.

Main Findings

• No agent clears 0.20. Across five frontier agents (kimi-k3, gpt-5.6-sol, qwen3.8-max, glm-5.3, deepseek-v4-pro), no agent reaches 0.20 out of a possible 1.0 at the default 45-minute budget. Scores range from 0.097 to 0.184.

• Difficulty calibration, not mode coverage, is the binding constraint. The in-band rate (deliveries whose target-model pass rate fell inside [0.125, 0.75]) ranges from 14.6% to 25.0%. Difficulty discards between 75% and 85% of deliveries before quality is consulted. Among the deliveries that survive, rubric coverage is high, from 0.700 to 0.967. Of the fourteen behaviours recurring across all five agents, eight bear on the pass rate and one on rubric coverage.

• Mode coverage is close to saturated. The term agents do well is quality, measured as coverage of a hidden rubric written by a fixed analyst (claude-opus-5) from the target model's transcripts. The analyst wrote 125 modes over 24 tasks, a mean of 5.2 per task, split into 86 failures, 20 detours and 19 handled points.

• The two failure modes are distinct, and one agent shows each. qwen3.8-max places the fewest deliveries in band (14.6%) but clears the gate 7 times out of 7 at quality 0.952. deepseek-v4-pro has the highest in-band rate (25.0%) and the lowest score (0.097), because only 5 of its 12 in-band deliveries survive the gate; its quality on survivors is 0.931. Placing a task near the original's pass rate cheaply means staying near the original, and the surface-swap defect is what catches that.

• Ranking is not stable across domains. No agent leads more than one suite. kimi-k3 is first on AutomationBench and fourth on Terminal-Bench; qwen3.8-max scores zero on AutomationBench and first on Terminal-Bench; glm-5.3 places in the top two on both of those and scores zero on Terminal-Bench-Science. The aggregate averages three capabilities that do not move together.

• Four times the time budget triples the score. Rerunning kimi-k3 at 180 minutes instead of 45 raises its score from 0.184 to 0.541, a factor of 2.9. Per suite, 0.31 to 0.56 on AutomationBench, 0.06 to 0.44 on Terminal-Bench, and 0.18 to 0.62 on Terminal-Bench-Science. The share of in-band deliveries clearing the gate rises from 90% to 100%, so the gain is not from restating originals.

• A usable task costs between $4.81 and $27.69, and the cost ordering disagrees with the quality ordering. kimi-k3 and gpt-5.6-sol score within 0.007 of each other (0.184 and 0.177), yet one costs 2.3 times the other. Per-episode wall clock is tightly clustered (41.2 to 48.4 minutes, within 18% of each other) while spend differs elevenfold; time per usable delivery varies twofold, from 202 to 410 minutes, driven almost entirely by how many deliveries survive.

• More budget buys output, not efficiency. At 180 minutes kimi-k3 spends 3.4 times as much and returns roughly three times as many usable tasks, leaving unit costs almost unchanged: $12.62 versus $11.88 per usable task, and 230 versus 244 minutes.

Methodology in Plain English

The unit of evaluation is an episode. The agent under evaluation receives one original task from a public suite plus a sanitised record of a fixed target model (deepseek-v4-pro) attempting that task six times, and must write one new task for the same suite in the suite's own directory layout — an instruction, an executable environment, and a verifier.

Three roles are separated. An analyst (claude-opus-5) reads the target model's transcripts and writes a hidden rubric of behavioural "modes" — general statements of what went wrong, or where a tempting alternative would have failed, stated broadly enough that a different task could provoke them. The author agent is the object of measurement and never sees this rubric. A judge (also claude-opus-5) later reads the delivered task and the target model's transcripts of attempts at it.

After the author agent delivers, the target model attempts the new task K times under the new task's own verifier. The judge runs only afterwards, because coverage is judged from what the task made the target model do, not from how the task reads.

Three terms multiply. The gate is zero if any of eight disqualifying defects is present (a verifier that does not check the result, an answer reachable without doing the task, an unsolvable task, a surface-swapped original, a copy of another task, a call to a disallowed model, a delivery that is not exactly one task, a duplicate within the run). One defect — the surface swap — is partly mechanical: the harness computes a word-level diff of the two instructions and mounts every differing span as fact; if every span is a renaming, the defect fires.

Difficulty is 1 if the target model's observed pass rate on the delivered task falls inside [0.125, 0.75], and 0 otherwise. With K = 6, that means the model solved the task at least once and at most four times out of six. No model judges this term; execution does.

Quality is not raw coverage but min(1, modes covered / ceil(aN)) with a = 0.6, so three modes out of a five-mode rubric earn full marks. The authors state that full coverage is the wrong target because one new task cannot stage every mode of its source without being that source — in an early run, every episode that reached full coverage was also gated as a clone.

Why This Matters

Impact on research. The paper argues that the field's evaluations of automated data synthesis measure a proxy (post-training gain or a trained checkpoint) rather than the delivered artifact. AutoDataBench closes that gap and, in the authors' framing, sits at the strong end of the verification hierarchy, because the delivered task's own executable verifier decides whether the target model solved it. It also supplies a measurement for a named underpopulated niche: governance-grade measurement of self-improvement.

Real-world applications.

• Producing training tasks for coding and terminal agents, where verifiable environments already exist and expert authoring is the bottleneck.
• Generating difficulty-matched exercises for scientific-computing agents, one of the three suites evaluated.
• Authoring cross-application business workflow tasks for enterprise automation agents.
• Benchmarking and procuring data-synthesis tooling on a per-task cost basis, since the paper prices one usable delivery in both dollars and wall clock.

Industry relevance. The paper's economics section is aimed directly at data companies and frontier labs. Delivered data is commissioned, paid for and accepted sample by sample, so a per-delivery cost metric is actionable in a way a post-training benchmark score is not. The domain-instability finding also has a procurement consequence: there is no single best author agent to standardise on, and the agent should be selected per domain rather than once for the whole corpus.

Future Directions

• Separate time budget from reasoning effort. The 180-minute arm was run at the agent's maximum reasoning-effort setting while the default arms used default effort, so the two variables move together. A clean attribution needs an arm at the default budget and maximum effort, which the authors have not run; they therefore read the comparison as a claim about direction rather than as a measurement of the factor.
• Test the assumption underneath the quality term. The paper states that what remains untested is the assumption that a task exercising a measured mode yields training data which repairs it.
• Improve difficulty calibration. Difficulty is the term an author agent cannot reason its way to, because the only way to learn where a draft's pass rate has landed is to run the target model on it and adjust — which motivates more efficient search over drafts.
• Broaden the task subset and domains. The subset is eight tasks per suite, so per-suite numbers are estimates over a fixed subset rather than over the suites as a whole; the authors note they have not run a controlled comparison of authoring behaviour across domains at scale.

Target Audience

Researchers and engineers working on synthetic data generation, self-improving agent systems, and reinforcement learning from verifiable rewards; data teams and platform owners who commission agentic training tasks and need per-artifact acceptance criteria; benchmark designers interested in evaluating artifact quality without downstream training; and practitioners choosing which frontier agent to deploy for authoring training data in a specific domain.

Authors’ abstract

Recent gains in language model capability have come more from data than from architecture. Frontier labs and data companies produce verifiable agentic tasks, which supervised finetuning and reinforcement learning then turn into capability.This production line still rests on human labour and on human-in-the-loop collaboration. Automating task creation would let data production scale with compute rather than with expert headcount, would extend to more domains, and would enable a key step in recursive self-improvement (RSI). Current evaluations of an agent's ability to write such tasks measure how a model performs after training on what the agent produced. That does not match common practice in the data industry, where data is delivered sample by sample and each sample is accepted against a set of criteria rather than put straight into training. No existing evaluation asks whether an individual task meets the acceptance criteria of a data pipeline. We therefore introduce AutoDataBench. Given an original benchmark task and a record of the target model attempting it, an agent must write a new task for the same suite that meets practical acceptance standards on validity, novelty, difficulty and behavioural coverage. Across three benchmarks of executable agent tasks, no agent we evaluate scores above 20 out of 100 at the default time budget of 45 minutes. Giving the strongest agent four times as long improves its score substantially, while the cost of one usable task stays almost unchanged. Current agents can write training tasks of the required quality, but not efficiently. AutoDataBench provides a direct measure of an agent's capacity for autonomous data synthesis: one artifact at a time, judged against the criteria a production pipeline would apply, and without a training run. Code and data are available at https://github.com/StarDewXXX/AutoDataBench.