AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

AI summary

Overview

Research area: AI agent reliability — automated failure diagnosis and evaluation for LLM-based agent systems.

Technical level: Advanced. The paper combines a benchmark and annotation study with a formal constraint-synthesis framework and multi-model empirical evaluation.

Scope: The paper introduces a 170-trajectory benchmark of annotated agent failures and AgentRx, a domain-agnostic framework that localizes the first unrecoverable failure step in an agent execution and assigns it a root-cause category.

What This Paper Is About

AI agents fail in ways that are hard to pinpoint because their executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. Manually figuring out which step caused an unrecoverable failure requires domain expertise and is expensive at the scale of agents serving millions of daily users. The paper's goal is to make that diagnosis automatic, evidence-grounded, and auditable, and to provide a public benchmark with human-annotated critical failure steps and failure categories.

Key Contributions

1. An open-source benchmark of 170 failed trajectories across 11 diverse task settings, each annotated with a critical failure step and a failure category, with a mean pairwise inter-annotator agreement of κ = 0.89.
2. AgentRx, a domain-agnostic diagnostic framework that normalizes heterogeneous multi-agent logs into a common representation, synthesizes constraints from tool schemas, domain policies, and observed trajectories, and emits a step-indexed validation log that an LLM judge uses to localize the critical failure and assign a category.
3. A cross-domain failure taxonomy with 9 root-cause categories, derived through grounded-theory coding rather than imposed in advance.
4. Extensive experiments reporting 75% average improvement in failure localization over prior work and 30% improvement in root-cause categorization over the authors' own judge baselines.

Main Findings

• Step localization improves over prior work. Against a modified Who&When baseline (W&W*), AgentRx reaches 42.7% ± 1.5 step accuracy on τ-bench (vs. 18.8% ± 5.3), 36.4% on Magentic (vs. 22.7% ± 3.9), 82.5% ± 1.4 on Flash (vs. 64.3% ± 3.9), and 61.5% ± 4.2 on RelWork (vs. 32.6% ± 1.3). W&W* does not predict a root-cause category at all.

• Category attribution is a capability prior work lacks. AgentRx reaches 47% ± 1.5 critical category accuracy on τ-bench, 46.2% ± 1.3 on Magentic, 65.9% ± 5 on Flash, and 59.3% ± 2.1 on RelWork.

• Violation evidence is the strongest single signal on some domains. On τ-bench, adding violation evidence (Baseline+Vio.) raises step accuracy from a 29.9% ± 1.2 baseline to 42.7% ± 1.5 and category accuracy from 30.8% ± 2.1 to 43.6% ± 1.2. Violations reduced average step distance on Magentic from 22.5 ± 3 to 14.9 ± 1.9.

• The taxonomy checklist helps where violations are sparse. On Flash, the checklist-only configuration reaches 82.5% ± 1.4 step accuracy, which the authors attribute to semantic structure outweighing sparse or noisy violation evidence; on Magentic, combining checklist and violations gives the best step accuracy (35.6% ± 1.3).

• Violations lift specific categories on τ-bench. Per-category results show System Failure going from 0.0 to 66.7 and Intent–Plan Misalignment from 5.6 to 61.1, while Instruction/Plan Adherence drops from 75.0 to 66.7 because the checklist-only judge uses it as a catch-all. Invention of Information, Intent Not Supported, and Under-specified Intent remain at 0.0 in both settings.

• Step-by-step constraint generation beats one-shot. The advantage is largest on the longest trajectories: on Magentic, step-by-step Checklist+Vio. reduces average step distance from 22.5 to 12.4, while on the shorter Flash trajectories the two strategies perform similarly.

• Cost is dominated by constraint generation. Under the GPT-5 pricing assumptions used, the judge-only baseline costs USD 0.05 per trajectory versus USD 0.18 for one-shot AgentRx, about 3.6× higher; constraint generation accounts for roughly 81% of total cost.

• The framework ports to unseen agent systems. Running without code changes on all 1,022 MAST trajectories produced a critical step, root-cause category, and supporting violation evidence for every trace, with 7.6% of traces receiving AgentRx labels with no MAST counterpart.

• Diagnoses can be misdirected. Qualitative analysis shows violation logs can point at the wrong failure type (a Flash System Failure case dominated by invocation-oriented checks) or at downstream symptoms (a Magentic-One case labeled Guardrails Triggered instead of the correct Invention of New Information).

• Relaxed metrics show the diagnosis is often near-miss. On τ-bench, AgentRx reaches 48.7% ± 4.2 at Acc@±1, 61.5% ± 3.6 at Acc@±3, and 68.4% ± 2.4 at Acc@±5, with an average step distance of 3.8 ± 0.6.

Methodology in Plain English

The authors first ran agents across several task settings, collected the runs that failed, and had three annotators read each trajectory step by step. Annotators used a grounded-theory open coding procedure in two phases: first marking every failure step with a short code and a reason, then deciding which failure was the earliest one the agent never recovered from — the "critical failure." Repeated coding was merged into higher-level categories until new trajectories stopped introducing new phenomena (theoretical saturation), after which the category definitions were frozen and remaining trajectories were re-coded under them.

From that analysis came 9 root-cause categories, and each trajectory in the benchmark stores its failure events (step index, description, label, rationale) and its critical failure.

For automation, AgentRx takes a toolset with schemas, an optional domain policy, and a failed trajectory. It normalizes logs into a common intermediate representation, then synthesizes two kinds of constraints: global constraints from the tool schemas and policy, and dynamic constraints from the task instruction and the trajectory prefix observed so far. Each constraint has a guard (does it apply at this step?) and an assertion (satisfied or violated) and is checked either programmatically over structured fields or semantically by an LLM. Violated constraints with their evidence go into a step-indexed validation log. An LLM judge then reads the task instruction, trajectory, taxonomy checklist, and validation log, and picks the first step whose violation evidence explains why the run fails, along with a category and a short rationale. The judge may override violations when trajectory context suggests otherwise, and can output "inconclusive."

The authors compare AgentRx against a judge-only baseline that sees the trajectory and taxonomy prompt but no validation evidence, and against Who&When adapted to their task definition of the first unrecoverable step. Each configuration is run n=3 times with mean ± standard deviation reported. gpt-5 is the default model, with o3 and DeepSeek-V3.2 results reported separately.

Why This Matters

Impact on research. The paper reframes agent evaluation as a debugging problem: not just whether an agent failed, but where recovery became impossible and why. It supplies gold annotations (critical step plus category) that prior agent benchmarks such as AgentBench, WebArena, and GAIA do not provide, and it shows that trajectory-level constraints are a usable signal for detecting failures. The benchmark and code are released publicly, so the taxonomy and attribution task can be tested on new agent systems.

Real-world applications:

• Diagnosing recurring incidents in production cloud services, the setting of the Flash workflow automation agent.
• Debugging tool-calling agents in retail customer service workflows, where agents cancel or modify orders, handle returns and exchanges, and update addresses under policy guidelines.
• Auditing generalist multi-agent systems that browse the web, navigate files, and execute code, such as Magentic-One.
• Reviewing software-development agents and open-ended web/file agents, the domains covered by the RelWork subset drawn from MAST and Who&When.

Industry relevance. The work comes from Microsoft Research and Microsoft, with a Hugging Face dataset and GitHub repository release. Because AgentRx is domain-agnostic and needs only a log-format adapter to port to a new agent system, it targets the operational problem of monitoring agents at scale: cheaper one-shot analysis for large-scale monitoring, more expensive step-by-step analysis for detailed debugging. The manual annotation cost reported — 62.5 human hours for 170 trajectories, at 20 minutes per τ-bench trajectory, 22 minutes per Flash and RelWork trajectory, and 24 minutes per Magentic trajectory — is the practical argument for automating the task.

Future Directions

• Extend the taxonomy to new domains. The authors state the current taxonomy may not cover all failure modes in other agentic domains and may require extension; adding a category requires only defining new checklist questions and adding them to the judge prompt.
• Reduce misdirection from noisy or downstream signals. The paper calls for identifying the smallest set of high-quality signals needed to separate true failures from noisy flags, and describes an optional reflection and pruning stage that removes likely false-positive constraints before judging.
• Improve the cost-accuracy tradeoff. One-shot constraint generation is cheaper (USD 0.18 vs. USD 0.05 per trajectory, with constraint generation roughly 81% of cost) but weaker on long trajectories; better allocation between cheap monitoring and detailed debugging remains open.
• Scale evaluation beyond the current benchmark. The MAST run over 1,022 trajectories was used as a generalization stress test rather than an accuracy benchmark because MAST provides no gold labels for the critical-step/root-cause task, leaving the question of how well the framework generalizes under gold supervision open. The paper does not report accuracy numbers for that run.

Target Audience

Researchers and engineers working on AI agent reliability, agent evaluation, and LLM-as-a-judge systems; developers operating production multi-agent or tool-calling workflows who need faster root-cause analysis; and anyone building failure taxonomies or annotated trajectory datasets. Readers need familiarity with LLM agents, tool calling, and evaluation metrics to follow the framework formulation, though the benchmark, taxonomy, and results are presented in a way that practitioners can apply directly.

Authors’ abstract

AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an $\textit{automated diagnostic framework}$ that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.