The Pulse
LHTB 2.0 Gives AI Agents Three Hours to Finish 77 Tasks
A paper posted to arXiv on October 9 introduces Long-Horizon-Terminal-Bench 2.0, a 77-task benchmark that tracks agent progress over three-hour runs. Across 15 models, the top system reaches the benchmark’s near-completion threshold on 37 t
AI.info Team · Written with AI ·
Only 192 of 1,145 scored runs—16.8%—passed overall in a new test of AI agents given up to three hours to complete extended workflows. The benchmark, Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0), scores not just whether an agent finishes, but how much verifiable progress it makes along the way. The paper was posted to arXiv on October 9 by a 17-author team.
Three hours, with checkpoints every 30 minutes
LHTB 2.0 expands its predecessor from 46 to 77 terminal-based tasks, covering 13 domains and eight task types. The suite includes software repair, scientific analysis, games, professional workflows and tasks that ask agents to reproduce the behavior of unfamiliar scientific or engineering tools. Agents work in containerized environments, where they can inspect files, edit code and run commands against a task’s stated objective.
The extended run budget is three hours, up from 90 minutes, with verifier checkpoints every 30 minutes. A checkpoint records whether work has advanced, stalled or lost ground; the test can therefore distinguish a nearly finished run from one that made little progress, even if neither meets the final pass threshold.
“On the 30 reconstruction tasks, binary verifier scores average 0.221, whereas dense ladder scores average 0.653, showing that outcome-only evaluation discards substantial intermediate progress.”
Zongxia Li et al., authors of “Long-Horizon-Terminal-Bench 2.0”
GPT-6-astra leads, but misses more than half
The team evaluated 15 models across 1,145 scored model-task runs. GPT-6-astra records the highest average dense reward, 0.69, and reaches the near-completion threshold of 0.95 on 37 of 77 tasks. That is a 48.1% near-completion rate; its exact-completion rate is lower, at 42.9%. GPT-5.6-sol ranks second, with a dense mean reward of 0.67 and 32 tasks at or above the near-completion threshold.
The overall ranking hides differences between task families. GPT-6-astra leads on the 30 reconstruction tasks, where it reaches a mean reward of 0.93 and the near-completion threshold on 24. On the 47 released-workflow tasks, GPT-5.6-sol comes out slightly ahead, with a mean reward of 0.57 against GPT-6-astra’s 0.54.
More time changes some results—and the bill
Longer runs matter for models that solve tasks slowly. Across the benchmark, 42% of successful runs first cross the near-completion threshold after the one-hour mark, and 21% do so after two hours. A one-hour cutoff would have moved Hunyuan hy4 from fourth place to ninth and DeepSeek V4 Pro from eighth to 13th, according to the paper.
Extra time also carries a measurable cost. The authors estimate an average of $48.41 per task across the models, using token counts and public list prices. Their estimate puts GPT-6-astra at $397.16 per task, versus $67.60 for GPT-5.6-sol; the paper says its figures do not include prompt-cache discounts. Between 83% and 99% of each model’s spending went to tasks it did not solve.
Reconstruction tasks test what agents can infer
Thirty of the 31 new tasks use a reconstruction method: a working scientific or engineering artifact is parameterized to generate scenarios, with some examples exposed to the agent and others held back for evaluation. The agent must infer the artifact’s behavior from the examples, then implement a solution that generalizes to unseen cases. A dense progress ladder awards credit for intermediate milestones rather than treating every incomplete run as having done nothing.
The researchers also describe changes meant to improve the reliability of their scores. They corrected task and evaluation-harness defects, isolated verifier runs from the active agent and checked successful trajectories for access to hidden test materials. Even with these measures and partial-progress scoring, only 16.8% of all scored runs pass overall.