Agent Evaluation Starts With Environment State

An agent evaluation should specify the starting environment, the task contract, the action boundary, and the state that counts as success. Final prose is one artifact. But many agent tasks are completed or failed somewhere else entirely: a reservation database, a code repository, a filesystem, a live web application.

Outcome grading must distinguish proposal, staged action, committed side effect, and verified postcondition. A plausible narrative cannot substitute for the state transition the task required. The benchmarks that survive scrutiny are built exactly this way. τ-bench compares databases. WebArena checks functional correctness inside replica websites. OSWorld ships a setup configuration and an execution script with every task. Every figure in this lesson comes from that kind of grading, not from a judge reading the closing message.

The itinerary that was never booked is not a thought experiment. It is the failure a published benchmark was built to catch. τ-bench puts a model into a retail or airline customer-service loop with tools, a simulated user, and a database sitting behind it. Sierra published it in 2024. Its grader never reads the agent's final message. “We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state,” the abstract says. Sierra's own write-up puts it the same way: “We used a stateful evaluation scheme that compares the database state after each task completion with the expected outcome”. The database is the verdict.

Graded that way, fluency stops paying. Even GPT-4o succeeds on under 50% of tasks. And the results do not hold still under repetition. On τ-retail, GPT-4o starts at a pass^1 of roughly 61% and falls to “~25% on pass^8 in τ-retail, which is a staggering 60% drop compared to its corresponding pass^1 score”. That is 60% in relative terms, not 60 percentage points. Fewer than one task in four survives eight independent attempts. A single run, graded on the answer, would have reported none of that.

An evaluation is defined by its Initial state, its Task contract, its Action trace, and its Terminal state. None of those is the final message.

The Initial state is a reproducible snapshot of accounts, files, pages, tools, and permissions; OSWorld ships one per task as a setup configuration. The Task contract fixes goal, constraints, authority, evidence, and acceptable non-success states. The Action trace records tool requests, results, approvals, retries, and errors. The Terminal state is the environment after the run, including partial or unintended effects — in τ-bench, the database, held up against the annotated goal state. The Grader is a deterministic or independently reviewed judgment over state and evidence.

A grader can be as mechanical as the sequence SWE-bench documents. It “Applies the model’s generated patch”. It “Runs the repository’s test suite”. It “Determines if the patch successfully resolves the issue”. Nothing in that sequence looks at the explanation.

The Terminal state and the Grader should keep owners of their own, and tests of their own.

Response grading, State grading, and Trajectory grading look at three different artifacts, and only one of them is the world. A judge that scores only the final text is useful for explanation quality. It misses external effects, and it is weak for transactional agents. A checker that compares the final environment with explicit success conditions gives direct outcome evidence. It needs reproducible fixtures, and it is the strong default. A reviewer who inspects actions and intermediate evidence finds unsafe shortcuts and can overreward stylistic traces. It belongs alongside outcome grading, not instead of it.

WebArena puts a number on the difference. It built fully functional replica websites and graded agents on the functional correctness of task completions, not on what the agent said it had done. “The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%,” the abstract reports. People doing the same tasks in the same environment finished 78.24% of them. The best GPT-4-based agent, measured on what actually happened on the site, finished 14.41%.

Grading holds up when the fixture behind it can be reset and rerun across models or policies. Every system under test then faces identical starting conditions. It stops holding up when a score can be earned by reaching the right answer through an unsafe route. The number then rewards the outcome and ignores how the agent got there.

One executable task, with a resettable fixture and a checkable terminal state, is worth more than a page of rubric. Write it so there is a shortcut to the correct answer that you would not want taken, and so the grader can tell which route was used. Then run the task again from a clean reset, and once more against a different model or policy. That is how the fixture itself proves it can be reused.

Five moves make the task executable. Freeze the starting world: create a deterministic fixture with known records, permissions, and tool behavior. State success precisely: define terminal facts, constraints, and disallowed side effects. Instrument the run: capture calls, responses, approvals, retries, and state versions. Build independent graders: use code for objective postconditions and calibrated review for subjective quality. Test alternate paths: include partial success, timeout, denial, and malicious observations.

OSWorld shows the first two of those moves shipped 369 times. Its abstract describes every task the same way: “Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation.” An initial-state config and a per-task execution grader, 369 times over, is the resettable-fixture discipline made concrete. Read the scores next to the design: “while humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success”. Fixtures this explicit are what make a gap that size legible rather than arguable.

Start with tasks that can be reset and graded deterministically. Add human judgment only where the desired quality cannot be represented by state or rules. When a score improves, ask first whether the gain came from a route you would refuse to ship in production. The whole approach is only as good as the fixture underneath it. If that fixture cannot be restored and replayed, a comparison between two runs means very little.

Execution-based grading is the reason SWE-bench is quoted at all. It assembled 2,294 problems from real GitHub issues and their pull requests across 12 popular Python repositories. The verdict is whether the repository's tests pass afterwards. Epoch AI describes the same procedure: “Once the model has made its changes, the solution is evaluated by running unit tests on the modified codebase”. Nobody grades the explanation. At publication that was punishing: “The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues.”

The Verified subset holds 500 of those instances, screened by people. Epoch AI records that “The dataset was curated through a rigorous human annotation process involving 93 software developers. Each sample was reviewed by three separate annotators to ensure the issue description is well-specified, the unit tests are appropriate, and the sample is free of other major issues”. Anthropic describes it the same way: SWE-bench Verified “is a 500 problem subset of SWE-bench that has been reviewed by humans to make sure they are solvable”.

Then comes the part that makes the fixture the real subject. A separate team went back and hand-screened the SWE-bench instances that SWE-Agent+GPT-4 had resolved. They published the audit as SWE-Bench+, and this is what it says: “1) 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. We refer to as solution leakage problem. 2) 31.08% of the passed patches are suspicious patches due to weak test cases, i.e., the tests were not adequate to verify the correctness of a patch. When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%.” The environment is the grader. Which is precisely why the environment is also the thing that has to be audited. 12.47% and 3.97% are the same agent on the same task set. Only the fixture changed.