Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

AI summary

Overview

Research area: Evaluation and benchmarking of AI agents, specifically the agent harness — the software layer (context management, tool interfaces, execution and permission controls, session lifecycle) that sits between an LLM and its environment.

Technical level: Intermediate to Advanced. The paper assumes familiarity with LLM agent architectures, tool-calling protocols, and benchmark design; the systems-engineering portions (trace normalization, evidence provenance, admission gating) are the most demanding.

Scope (one sentence): The paper introduces TraceTrial-Bench, a 95-task, seven-component benchmark with trajectory tracing, multi-source evidence collection, and failure-attribution tooling that scores agent task performance and harness reliability separately across 4 harnesses and 9 models (3,420 runs).

What This Paper Is About

Most agent benchmarks judge only the final answer, test result, or environment state, which conflates two different things: whether the model made good decisions and whether the harness faithfully delivered information, executed commands, and enforced permissions. The authors argue that an agent is "Model + Harness," so a passing task can hide a broken harness mechanism and a failing task can occur despite a correct harness. TraceTrial-Bench is built to separate these: it runs mechanism-specific tasks inside each harness's native environment, records full execution traces, and produces two independent score sheets plus an evidence-based account of how a harness failure may have propagated into an agent failure.

Key Contributions

1. A mechanism-oriented benchmark. TraceTrial-Bench contains 95 expert-level tasks organized under seven harness reliability components — Context (15 tasks), Execution (27), Governance (9), Lifecycle (19), Observability (7), Tools (17), and Verification (1) — each pairing a harness obligation with agent deliverables and an executable or state-based ground truth.
2. A two-layer evaluation framework. A Trajectory Tracing (TT) module normalizes native execution records into a shared Trace Intermediate Representation (Trace IR), a Multi-source Evidence Acquisition (MEA) module builds bounded, criterion-specific evidence packets from traces, workspace artifacts, simulator state, and probes, and the two layers are scored separately with the formula S = 50·(p_H/n_H) + 50·(p_A/n_A).
3. Failure attribution through explicit mechanism chains. Tasks can pre-declare a link between a harness criterion and an agent criterion plus an activation condition; links activate automatically when both endpoints fail, and are distinguished from validated propagation, which requires manual review of event order, an observable intermediate distortion, and competing explanations.
4. A reproducible release. Tasks, frozen configurations, traces, verdicts, manifests with digests, and attribution outputs across 4 harnesses (Codex, Claude Code, OpenClaw, DeepSeek Harness), 9 models, and 95 tasks — 3,420 selected executions — are provided for audit and replay.

Main Findings

• Harness scores fall well short of perfect even with strong models: macro-averaged harness scores across the evaluated harnesses range from 59.8% to 72.0%.
• Task success conceals harness unreliability: 31.1% of runs achieve task success despite failing at least one harness criterion.
• Rankings depend on what you measure: task success and harness-criterion satisfaction produce different harness rankings, so outcome-only leaderboards would select a different harness than a reliability-focused evaluation.
• Same model, different harness behavior: because each harness retains its native prompt construction, tool transport, context policy, permissions, and session handling, identical model–task combinations can be compared across harnesses without normalizing away the behavior under test.
• Failures have multiple distinct operational states: the decision table yields six stored labels from seven ordered rules — including inconclusive_device, inconclusive_unmeasured, harness_caused, agent_caused, and harness_defect_absorbed — so co-occurring harness and agent failures are not automatically reported as propagation.
• Measurement integrity is treated as a first-class concern: apparatus admission failures are explicitly not counted as harness defects, and empty evidence selection yields a zero that indicates "not established," not a proven defect.
• Per-harness scores, per-model results, and per-dimension breakdowns are not reported in the provided excerpt beyond the 59.8%–72.0% range and the aggregate statistics above.

Methodology in Plain English

The researchers treat the harness itself as the system under test and hold everything else fixed. For each task they write down, before running anything, what the harness must do (e.g., recover from an oversized tool output, preserve a fact across a session boundary, deliver a pending job's completion, redact a secret), what the agent must deliver, and an independent source of truth the agent cannot touch — a private simulator ledger, a frozen workspace baseline, or post-run probes.

Each run happens in a task-scoped container with a frozen starting workspace. Adapters read each harness's own logging interface (Codex's JSON event stream, Claude Code's stream-json plus session record, OpenClaw's JSON output plus session JSONL, DeepSeek Harness's JSON-RPC notification stream) and map the records into a common event schema with a shared envelope and a contiguous sequence index, while keeping the original raw record so the normalization can be audited. Tool calls and results are linked by the harness's native call identifier; ambiguous or duplicate records stay explicitly unlinked rather than being guessed.

Before any scoring, an admission gate checks that run identity, configuration, lifecycle records, archive manifests, baselines, probes, and private oracles are all consistent. Deterministic criteria are resolved by task-specific verifiers first and frozen; only then does a semantic judge see the remaining criteria, one at a time, each with a bounded evidence packet produced by a declared selector. Agent criteria are judged first using changed artifacts; harness criteria are judged afterward using traces and probes, with agent verdicts supplied only as downstream context — never as evidence that the harness failed. Failed harness and agent criteria are then matched against pre-declared mechanism chains to produce candidate links, which must be manually validated before being called propagation.

Why This Matters

Impact on research. The paper moves agent evaluation from a single outcome number toward a decomposed, auditable measurement of the whole system. Because it preserves native harness behavior and records every verdict with its evidence packet, it enables controlled studies of how harness design interacts with model behavior — a variable that outcome-only benchmarks systematically hide.

Real-world applications:

• Agent platform selection: teams choosing among coding or computer-use harnesses can compare reliability per component rather than trusting public task-success leaderboards.
• Incident diagnosis in production agents: the trace representation and failure-attribution chains give a template for distinguishing "the model chose badly" from "the harness lost the tool result" in deployed systems.
• Compliance and governance review: the Governance and Verification components plus digest-bound evidence support audits of approval flows, secret redaction, and whether an agent's completion claims match authoritative state.
• Regression testing during harness development: the seven-component taxonomy turns vague reliability goals into concrete, testable obligations with observable triggers and verdict conditions.

Industry relevance. The finding that 31.1% of successful runs violate at least one harness criterion is directly actionable for anyone shipping agents: outcome metrics alone will pass releases that silently drop context, misreport command status, or bypass guardrails. The four-level claim hierarchy (verdict, confirmed defect, activated link, validated propagation) also offers a vocabulary for reporting reliability without overclaiming causality.

Future Directions

• Expand coverage of thin dimensions. Verification has only 1 task and Observability 7, so conclusions about those components rest on very limited evidence; extending the suite would test whether the observed ranking divergence holds per component.
• Automate propagation validation. Candidate links currently require manual review to become validated claims; scaling or semi-automating that review (while preserving the event-order and intermediate-distortion requirements) is the obvious next step toward causal attribution at corpus scale.
• Separate model effects from harness effects more finely. The paper reports scores aggregated over 9 models and 4 harnesses; a per-model, per-harness decomposition would show whether the 59.8%–72.0% spread is driven by specific mechanisms or by model-specific interaction effects.
• Broaden the harness and task population. The current results cover 4 harnesses and 95 tasks with 3,420 selected runs; adding harnesses, non-coding domains, and longer-horizon multi-session workflows would test whether the taxonomy's obligations transfer.

Target Audience

Agent-platform engineers and harness developers who need to know which reliability mechanisms are actually working; benchmark and evaluation researchers interested in decomposed, trace-based scoring and failure attribution; and technical product or infrastructure leads who make deployment decisions based on agent benchmark results and need to understand why task success alone is an insufficient signal. Readers without exposure to agent architectures or trace-level instrumentation will find the systems sections dense.

Authors’ abstract

LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.