Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks

AI summary

Overview

Research area: Evaluation and synthetic data generation for LLM agents operating in command-line (terminal) environments; specifically, how to measure and construct "long-horizon" tasks.

Technical level: Advanced — the paper formalizes task length with definitions drawn from parallel computing (work and depth), describes a multi-stage generation pipeline, and reports controlled auditing experiments across seven datasets.

Scope (one sentence): The paper argues that existing long-horizon terminal benchmarks and synthesis pipelines mostly add work (number of operations) rather than depth (turns that must happen in sequence because each depends on reading the previous output), proposes a turn-capped measurement of depth, audits 734 tasks from six datasets, and builds a 105-task dataset called DEPTH that is deeper than all of them.

What This Paper Is About

Terminal tasks for agents are usually described as "long" by counting the turns a model happens to take, and as "hard" by difficulty labels an LLM assigns during generation. The authors show that turn counts mostly reflect a model's batching habits and that generation-time labels barely track how often models actually pass. They instead measure a task's depth — the smallest stated turn cap under which it can be solved — and then build new tasks by having a generation agent actually carry out each segment inside a real software environment before writing it down, so that later segments genuinely depend on what earlier ones produced.

Key Contributions

1. A measurement and an audit. The paper defines a terminal task by work W* (operations needed when the environment state is known, estimated by the number of lines in the reference solution) and depth D* (turns that must occur in sequence), estimates depth empirically by turn-capped evaluation, and uses it to audit 734 tasks sampled from six datasets (TMax, FACET, SETA, RST, Terminal-Bench 2.0, TerminalWorld-Verified).
2. Evidence that depth, not turn count, predicts difficulty. Once work is controlled, deeper tasks have lower public pass rates, while turn count is unrelated to pass rate once depth is known; depth also agrees with human author ratings in TB2 (+0.55 rank partial correlation).
3. A construction method and dataset (DEPTH). A generation agent explores a seed environment, writes each segment's instruction, performs the segment in the container, and then writes the solution and verifier from what the software actually did; the resulting 105 tasks exceed every audited dataset in both depth and work, and none of six evaluated models passes half of them.
4. A training result. Fine-tuning Qwen3-8B on 4,941 samples built from 1,272 runs on these tasks raises pass@1 on Terminal-Bench 2.1 (2.6% to 4.9%) and OpenThoughts-TBLite (3.4% to 9.1%).

Main Findings

• Synthesis has added work, not depth. Of the 296 solved synthetic tasks, 278 (94%) have D ≤ 3. In TMax and SETA, nearly nine in ten tasks have D = 1 — a model passes by issuing every command in the first turn without reading any output.

• Turn counts overstate depth by several times. TMax and FACET report 16.3 and 11.86 mean turns, while the median D of their solved tasks is 1 to 2. On the 88 solved TMax tasks with released Gemini runs, Gemini takes a median of 8 turns while the median D is 1. Across the six datasets, 96 of the 170 tasks with N between 6 and 10 have D = 1.

• Synthetic tasks are not deeper despite more work. Median W of solved tasks is 68 lines for synthetic datasets versus 17 for human-curated ones, yet the share with D ≥ 5 is no higher (6% versus 10%). RST, the deepest synthetic set, still has 79% of solved tasks at D ≤ 3. Human-curated sets hold a small deep tail: 11 of 105 solved tasks have D ≥ 5.

• Depth predicts difficulty beyond work. Controlling for W, the rank partial correlation of public pass rate with D is −0.31 on the 105 solved TB2 and TW tasks and −0.49 on TB2 alone. On the authors' tasks it is −0.29 (95% CI −0.53 to −0.02) over 52 solved tasks, and −0.31 when pooled.

• Turn counts carry no signal once depth is known. Controlling for D, N is unrelated to public pass rate on TB2 and TW (+0.06, 95% CI −0.18 to +0.26).

• Generation-time difficulty labels barely track pass rates. Labels set during generation correlate with released pass rates at only −0.07 to −0.11, versus −0.46 for TB2's human author ratings. Nine of the 11 sampled TB2 tasks rated hard have D ≥ 2, against 8 of the 28 rated easy or medium.

• Only tasks large in both dimensions are harder. Splitting TB2 and TW at D = 2 and the median W, tasks large in both have a mean within-dataset pass-rate percentile of 0.29 (lower is harder), versus 0.53 to 0.59 for the other three groups.

• The new tasks are deep and large. All 105 tasks come from different seed environments and have three to six segments, six for most. DeepSeek-V4.1-Flash solves 54, and 44 of those have D ≥ 40. Median W is 315 lines (above the 90th percentile of existing synthetic tasks at 226), and median N is 45 turns versus 5 to 9 for existing datasets.

• Depth is not an artifact of segment delivery. Merging the segments of 25 randomly sampled tasks into a single instruction leaves D unchanged for 68% of them; most drops are by only one cap.

• Depth is not inherited from the seed environment. In 35 pairs sharing an environment, the released original tasks have D ≤ 10 in 34 pairs (D = 1 in 26), while the authors' tasks have D ≥ 20 and the higher D in all 35 (sign test, p = 5.8 × 10⁻¹¹).

• Verifiers see the key steps. Replaying solutions with parts removed, a segment's checks fail before it is done in 601 of 602 segments (99.8%) and fail when its key step is removed in 598 of 604 (99.0%); with the key step removed, the next segment also fails in 421 of 500 cases (84.2%).

• Models are separated, and frontier models fail most tasks. pass@1 ranges from 17% (GLM-5.3-Flash) to 47% (Claude Sonnet 5.5), a 30-point spread, with GPT-6.1 Sol at 44%. Of the 90 tasks run by all six models, 29 are passed by none. Per-segment reliability ranges from 0.88 (Claude Sonnet 5.5) to 0.74 (GLM-5.3-Flash), a gap that compounds over six segments into 47% versus 16% of runs passing all of them.

• Generation is cheap. Generation cost $113.66 for the 136 seed environments, yielding 105 accepted tasks (77%), about $1.08 per accepted task.

Methodology in Plain English

Two ways to be long. The authors borrow from parallel computing: a task's work is how many operations it takes if you already know the environment, and its depth is how many turns must happen one after another because you can only choose the next command after reading the last output. A task with 200 independent operations can be one turn; a task where an unfamiliar version's default config hides a problem and fixing it exposes another is many turns.

Measuring depth. They state a turn cap in the instruction and stop the agent when it is reached, using the caps 1, 2, 3, 5, 10, 20, 40 and 80. The smallest cap under which a task passes is its estimated depth. Because a model that batches commands well can make a task look shallower, they take the minimum over two models (DeepSeek-V4.1-Flash and GLM-5.3-Flash) and note that adding the second model lowered D for 25% of existing tasks and 7% of their own solved tasks. Work is estimated simply as the number of non-blank, non-comment lines of the reference solution.

Building depth instead of assuming it. Seed environments come from public terminal-task packages, mostly the 1,530 TerminalWorld packages; after filtering and LLM judging, 136 seed environments were kept. A generation agent then works inside the container: it explores, picks one overall goal a real user would pursue, and splits it into segments. For each segment it writes the instruction, executes the work, and only then writes the reference solution and the checks — so the checks reflect what the software actually did. A clean container replays the solutions to confirm the state chain holds. Each segment contains at least one "key step" where the usual approach silently or loudly fails and only running the commands reveals it. Segment s+1 must consume a result that exists only after segment s's key step is done, which forces the evaluated agent to read output and change course. The final segment does one operation on the whole system, and earlier results must survive it.

Verifier rules. Checks ask the running system through its own tools (a client queries the service, a validator accepts the config) rather than looking for specific commands; they must fail both before the segment and after a solution with the key step removed; and every path, port or account a check names must appear verbatim in the instruction while the fixes must not. A task is accepted only if replaying the full reference solution in a fresh offline container passes every segment.

Evaluation and training. Six models each run every task twice without a cap in the terminus-2 harness, stopping at the first failed segment; pass@1 is averaged within tasks then over tasks. For training, runs are kept up to the last consecutively passed segment, cut wherever terminus-2 would compress the context, and continued from the compression summary so the model learns to resume after context compression.

Why This Matters

Impact on research. The paper argues that "long-horizon" should be a measured property, not an assumed one, and recommends that benchmark and training-data builders report depth alongside work. It also shows that the widely used proxies — turn counts under a loose limit and LLM-assigned difficulty labels — can mislead: 94% of solved synthetic tasks collapse within three turns, and generation-time labels correlate with actual pass rates at only −0.07 to −0.11.

Real-world applications.

• Agent evaluation: scoring suites that measure whether an agent can sustain a long interaction with an environment rather than execute a pre-planned script.
• Training data for small open models: the Qwen3-8B result shows trajectories from segmented, environment-grounded tasks transfer to single-instruction benchmarks (2.6% to 4.9% on TB2.1; 3.4% to 9.1% on TBLite).
• Configuration and operations troubleshooting: the key-step design mirrors the real failure mode of deployed systems, where version defaults, old command-line interfaces or truncated files cause failures that only show up when commands are run.
• Verifier design: the three rules for writing checks (outcome-based, tied to the key step, stated in the instruction) are a reusable recipe for any pipeline that needs tests that cannot be trivially satisfied.

Industry relevance. Generation costs about $1.08 per accepted task, which makes environment-grounded task synthesis a practical alternative to scarce human curation, and the segment-by-segment construction maps directly onto how agents are deployed — maintaining state across a long session rather than answering one instruction.

Future Directions

• How far does depth generalize beyond the terminal? The measurement depends on a stated turn cap and on the specific harness (terminus-2, with its reply-length, 10,000-byte output truncation and wait limits); whether D transfers to other harnesses, other action spaces or GUI agents is not tested.
• Does the training gain scale? Only Qwen3-8B was fine-tuned, on 4,941 samples from 1,272 runs, and gains on TB2.1 were small in absolute terms (2.6% to 4.9%). Larger models, more data, or reinforcement learning on these tasks remain open.
• Can depth be measured more cheaply? D is an empirical estimate that requires running models at up to eight caps, and tasks no model solves get no estimate at all — a limitation explicitly acknowledged. A cheaper proxy for D* would make the audit practical at benchmark scale.
• Can depth be synthesized in other domains? The pipeline depends on real software whose undocumented behavior can be discovered by execution; whether the same "carry out before writing" principle produces depth in domains without such version-specific quirks is unaddressed.

Target Audience

Researchers and engineers who build agent benchmarks, synthesize agent training data, or train agents for computer and terminal use; evaluation leads who need to justify that a benchmark is genuinely long-horizon; and readers interested in verifier and task-generation methodology. The formal definitions (work, depth, strategies, turn-capped evaluation) and the audit statistics make it most useful to those with some background in agent evaluation, though the central argument — that many turns and many operations are not the same as required sequential dependence — is accessible to a general machine learning audience.

Authors’ abstract

Long-horizon terminal tasks are used to evaluate and train agents. Human-curated datasets are scarce, so many synthesis methods have emerged. They usually measure the length of a task by the turns a model needs to pass it, and its difficulty by labels the LLM sets during generation. We find, however, that the turn count largely reflects the habits of the model, not only the demands of the task, and that the labels barely track how often models pass. We describe a terminal task by its work, the number of operations it requires, and its depth, the number of turns that must happen in sequence because the commands of each turn can only be chosen after reading the output of the previous one. Estimating depth as the smallest turn cap, stated to the agent, under which the task passes, we audit 734 tasks from six existing datasets. Synthesis has mostly added work, not depth: 94% of the solved synthetic tasks pass within three turns, yet depth, unlike the turn count, predicts difficulty beyond work. We propose DEPTH, in which a generation agent writes each task one segment at a time in a real software environment, carrying out each segment before writing it and starting the next from the state it leaves. Our 105 tasks exceed every audited dataset in depth and work, and none of six current models passes half of them. Fine-tuning Qwen3-8B on runs from these tasks improves it on two terminal benchmarks.