Skip to content
AI.info

The Pulse

METR finds AI models usually match or beat skilled humans on its coding tasks, with time horizons doubling every 3.6 months

METR found that tested models usually matched or exceeded skilled humans on its coding and optimization tasks. Its updated evaluation also found weaker model judgment and difficulty finding human testers who beat them.

METR finds AI models usually match or beat skilled humans on its coding tasks, with time horizons doubling every 3.6 months

The models METR tested usually matched or outperformed skilled humans on its coding and optimization tasks. The November 20 evaluation covers public models up to Opus 5.5 and GPT-6 Astra and also finds areas where AI agents appear to exercise worse judgment than human experts.

On a combined set of tasks, METR estimates that model time horizons are doubling roughly every 3.6 months. The measure tracks the length of tasks models can complete, using human completion time to assess task difficulty.

METR remains uncertain both about how quickly capabilities relevant to AI research and development are improving and about which capabilities would be needed for drastic acceleration. It says public evidence is insufficient to rule out such acceleration in the next couple of years, defining it as progress so fast that humans can no longer meaningfully understand, assess and respond to catastrophic risks from new AI systems.

Finding human comparisons

When METR recruited its own human testers, it says it typically failed to find people who outperformed the latest models. In a couple of tasks with existing records of at least 10 domain specialists attempting them, models tended to rank at a very high percentile, though not at the absolute top.

When METR first released its time-horizon work in March 2025, the best model measured was Sonnet 3.7, at around one hour. Within a year, models had saturated the TH1.1 task suite, which includes tasks taking up to 30 hours.

For the new work, the group recruited more experienced human testers than the STEM graduates used previously, but says the expertise and time required made recruitment difficult. Human baseline results were directly used for only 9% of its estimates of how long a human would take to complete a task; other estimates drew on public records, experts and extrapolation. Testers were allowed to use Sonnet 5 for implementation.

Judgment and task design

On tasks requiring choices without clear empirical feedback, AI agents appeared to have worse judgment than human experts, METR says. In one task, model judges overestimated the quality of an output made by a similarly strong or stronger model when compared with an expert scorer.

MIT Technology Review reports that one task asked models to design AI evaluations for METR, with human employees scoring the results. None of the AI-designed evaluations was good enough for employees to consider useful for their research, the outlet reports.

The group estimated that models performed somewhat worse on tasks costlier to turn into reinforcement-learning training environments, even after accounting for estimated human completion time. It used a model-estimated measure called lifecycle cost to quantify the expense of making, running and scoring evaluations.

A doubling of lifecycle cost for a task of a given length imposed a difficulty penalty very roughly equivalent to halving an agent’s time horizon, METR estimates. The group observed similar rates of progress on its higher- and lower-cost task subsets, including improvement on its most costly tasks.

Limits of the trend

METR is concerned that easy-to-run evaluations could overestimate real-world capabilities as similar environments become common in training. Its newer tasks target particular strengths and weaknesses relevant to AI research and development, but cover a narrower range of general software-engineering work than TH1.1.

The combined time-horizon result uses TH1.1 and the more difficult TH2 suite. METR says models have a lower 50% time horizon on the combined suite than on TH1.1 alone; a shorter doubling trend on a verifiable subset of TH2 could indicate a recent acceleration hidden in the combined trend, though other explanations are plausible.

A separate composite measure, the METR Capability Index, did not show strong acceleration or deceleration across the public frontier models tested. METR found uncertain evidence of changes in some developer-specific trends, but cautioned that those findings were sensitive to methodological choices.

Sources

  1. Original source · Blog post METR metr.org

Other pages read

  1. MIT Technology Review technologyreview.com

In the tracker

Related stories

Explore

More articles