Reliability, Simulation, and Benchmark Design

Agent reliability is a distribution over tasks, environments, model samples, and operational conditions. A credible evaluation asks four things at once. How often does the agent succeed? How badly does it fail when it fails? How much does the result move between runs? How sensitive is it to a change in the setup? One showcased trajectory answers none of them.

Two benchmarks show what that distribution looks like once somebody measures it. WebArena, published in 2023, is a reproducible environment of self-hosted e-commerce, forum, code-collaboration and CMS sites. Its authors “curated 241 templates and 812 instantiated intents”. Then they reported the score: “The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%.”

OSWorld took the same idea down to the operating system in 2024. It is “a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications”, running on Ubuntu, Windows and macOS. Each task carries its own initial-state setup and its own execution-based grading script. Its result: “While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge.”

14.41% against 78.24%. 12.24% against over 72.36%. No impressive trajectory prepares anybody for gaps that size. Public benchmarks such as GAIA, WebArena, OSWorld, SWE-bench, and tau-bench test useful capabilities, but product release decisions still require local tasks, policies, tools, and failure costs.

The demo used a clean account, a short task, and one lucky sample. “One lucky sample” is not a figure of speech. It has been measured, under settings chosen to make luck impossible.

Five LLMs, every one of them configured to be deterministic. Eight tasks. Ten runs each, in zero-shot and few-shot settings. The 2024 study behind those numbers, by Berk Atil, Breck Baldwin and eleven co-authors, summarises itself in one sentence: “We see accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%.” No model in the study delivered repeatable accuracy across all tasks, let alone identical output strings.

That is the demo and the next morning, before production adds any difficulty of its own: stale pages, ambiguous names, rate limits, tool errors. Under identical settings, the identical prompt has already moved by up to 15 accuracy points. And the distance between the best run you could have shown and the worst run a user could have met reaches 70.

A demo suite, a scenario benchmark, and a simulation program answer questions of very different sizes, at very different costs. A demo answers whether it can work once. A simulation answers how often it works, and at what price.

A demo suite is a few hand-picked tasks run once: fast to iterate on, heavily selection-biased, not release evidence. A scenario benchmark is versioned tasks sampled from known categories. WebArena is the published shape of one — “a ready-to-use benchmark with 812 long-horizon web-based tasks”, built from 241 templates. The results are comparable across systems, at the cost of going stale as models absorb them. A simulation program is many generated or replayed environments with controlled perturbations. OSWorld shows what that costs to build: 369 tasks, each with its own initial-state setup and its own execution-based grading script, across three operating systems. Broad stress coverage, useful for rare events, and a fidelity claim that has to be validated rather than assumed.

A number is worth quoting when it comes from repeated evaluation that shows the variance a single run hides — τ-bench's pass^8 rather than its pass^1. It is worth distrusting when the gain may have come from getting better at the benchmark rather than at the job it stands for. On SWE-bench, 12.47% became 3.97% once the leaked solutions and the weak tests were removed.

Reliability is a claim about a task distribution, repeated trials, environment variation, and a severity model, rather than about one run. Representative easy, typical, rare and adversarial cases. Several stochastic runs for each meaningful case. Latency, stale state, missing data, tool errors and interface changes. And a severity model that separates a harmless retry from a privacy or financial incident. The severity model and the uncertainty report should have separate owners, and separate tests.

The repeated-trials node has a metric behind it, and a published number. τ-bench, released by Sierra in 2024, defines pass^k as the chance that all k independent trials of the same task succeed. It measures that over 115 retail and 50 airline tasks. The abstract: “Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail).” Later the paper states the collapse in the form a release decision needs: “Even for the best-performing gpt-4o function calling agent which has a > 60% average task success, pass^8 drops to < 25%”.

Roughly 61% at pass^1. Under 25% at pass^8. Same agent, same tasks: one number for the demo, a different number for the week that follows. Run the same task many times and report the spread, not the best run.

Plan the reliability study before the next release, using the workflow that release will touch. Define the population: the users, tasks, environments and frequencies the suite claims to represent. Stratify the scenarios so routine work, boundaries, rare failures and malicious inputs are all present. Include cases the agent was never tuned against, so that progress made on the test itself cannot pass as progress on the job. Weight failures by consequence rather than count alone. Then publish the intervals, the exclusions, and the scenarios with insufficient evidence.

The repetition step is the one that usually gets a shrug and the word “enough”. A 2024 Anthropic paper on adding error bars to evals replaces the shrug with arithmetic. Take binary scores whose conditional means are uniformly distributed — uniform question difficulty. Evan Miller works it out: “Going from K = 1 (no resampling of answers) to K = 2, the total variance is reduced by 1/3. Increasing to K = 4, we have a variance reduction of 1/2, and setting K = 6, we reduce variance by 5/9.” The reduction has an upper limit of 2/3. Resampling buys a bounded amount of certainty. The rest has to come from somewhere else.

It cannot be bought more cheaply, either. One section of the paper carries the instruction as its own title — “Don't touch the thermostat!” — and the rule under it is flat: “In neither case should the sampling temperature be adjusted for the sake of reducing variance in the scores.” A tighter interval bought by turning the temperature down is a description of a system your users will never run.

For high-impact agents, publish both average success and worst-case or high-severity failure evidence. A single aggregate score can hide unacceptable pockets of risk. Before quoting a reliability figure, check where the improvement came from, since benchmark progress can come from learning the benchmark rather than the underlying job. Then publish the repeated runs behind the figure, so a reader can see the variance instead of taking one result on trust.

This is not editorial preference. NIST wrote it into a voluntary standard in January 2023, in its AI Risk Management Framework. MEASURE 2.3 asks that “AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s). Measures are documented.” MEASURE 2.5 goes to the boundary of the claim: “The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the conditions under which the technology was developed are documented.” And the framework rules out the naked percentage: “Accuracy measurements should always be paired with clearly defined and realistic test sets – that are representative of conditions of expected use – and details about test methodology”.

A figure with no test set, no conditions and no stated limits of generalisability is not a reliability result. It is a screenshot.