Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

AI summary

Overview

Research area: AI agent reliability, failure diagnosis, and LLM-based evaluation (root-cause attribution for autonomous agent executions).

Technical level: Intermediate — assumes familiarity with LLM agents, benchmark evaluation metrics (F1, accuracy), and agentic tool use, but the core idea is conceptually simple.

Scope: This paper reframes automated root-cause attribution (RCA) of agent failures as an evidence-search problem and introduces "Continual Search," a multi-turn prompting framework that improves diagnosis on long-horizon execution logs.

What This Paper Is About

When AI agents run long tasks, they produce enormous execution logs (often hundreds of thousands of tokens). When something goes wrong, someone has to figure out what failed first and why — a task called root-cause attribution. Existing automated methods use an LLM "judge" that reads the trace once and picks a plausible-sounding culprit, often missing sparse, scattered evidence buried elsewhere in the log. This paper argues that RCA is fundamentally a search problem and proposes an iterative framework that keeps pushing the judge to examine evidence it hasn't looked at yet.

Key Contributions

1. Continual Search framework. An iterative, multi-turn method that prompts an agentic LLM judge to challenge its current diagnosis and actively search for unresolved or previously unexamined evidence, rather than merely re-confirming its answer.

2. MegaRCA-Mix benchmark. A new testbed of 50 human-annotated failure trials drawn from Harbor Index execution logs (median 286K tokens, ~1.05 MiB per trial), spanning 17+ benchmarks and including full evaluation records: trajectories, configs, verifier outputs, and sandbox artifacts.

3. Evidence that RCA is search-limited, not reasoning-limited. Experiments across five judge models and three reasoning-effort settings show that lower-tier models with Continual Search can match or beat frontier models, and that low reasoning effort often performs as well as or better than high/max effort.

4. Demonstration of benchmark diagnosis. The same framework turns a general failure-taxonomy classifier into a benchmark-auditing tool that outperforms a purpose-built diagnostic pipeline (ABA) at flagging defective tasks.

Main Findings

• Continual Search wins on long-horizon traces. On MegaRCA-Mix, GPT-5.5's F1 rises from 0.349 (single-turn) to 0.498, versus only 0.401 under passive continuation; Opus-4.8 rises from 0.478 to 0.620 (vs. 0.559 passive). TRAIL and TELBench show the same pattern.

• Gains track new evidence, not just extra turns. Deduplicated observation tokens grow from 42K to 58K under Continual Search, but only to 48K under passive continuation. Artifact-level coverage climbs from 70.8% to 97.4% by turn 4, while passive continuation plateaus at 77.0% — mostly by unlocking secondary files like configs and system logs.

• Short trajectories see no benefit — and often regress. On AgentRx (7.2K tokens) and Who&When (2.4K tokens), the first turn already covers nearly all evidence. GPT-5.5 drops from 0.345 to 0.241 on AgentRx and from 0.548 to 0.452 on Who&When, consistent with prior work on judges flipping verdicts under conversational pressure.

• Search beats scale. Sonnet-5 reaches a turn-4 Weighted F1 of 0.518 on TRAIL, comparable to Fable-5 while using ~76% of the compute. Opus-4.8 beats Fable-5 at lower cost. Low-effort reasoning often matches or exceeds max-effort.

• Sequential search beats independent resampling. On TRAIL, Continual Search (Opus-4.8: 0.205 Joint Acc., 0.539 Weighted F1) outperforms both self-consistency (0.138 / 0.430) and a four-model heterogeneous judge panel (0.173 / 0.431). Self-consistency actually underperforms the single-turn baseline despite consuming more compute.

• Benchmark auditing improves sharply. With Continual Search, a taxonomy classifier on MegaRCA-Mix jumps from an F1 of 0.16 at turn 1 to 0.74 at turn 3, surpassing the dedicated ABA pipeline (0.63). Gains come almost entirely from recall (0.09 → 0.61) while precision stays above 0.90.

• Results are stable. Across four independent Opus-4.8 judge runs on TRAIL, Continual Search beats passive continuation on both metrics every time.

Methodology in Plain English

The researchers start by running each benchmark's standard single-turn RCA prompt with a tool-enabled LLM judge (built on the Claude Agent SDK, with read-only access via Bash, Grep, Glob, etc.). That produces an initial attribution.

From that starting point, they branch the same session into two conditions for turns 2–4:

• Passive Continuation (control): the judge is asked to reconsider and reconfirm its answer without being told to look for new evidence.
• Continual Search: the judge is explicitly instructed to challenge its standing conclusion and search for evidence it has not yet examined — unread tool outputs, unexplored regions of the log, or unconsidered candidate failure steps.

Because the evaluation records are too large to inline into a context window, judges inspect artifacts on demand through tools rather than receiving the whole log at once. The authors then measure attribution accuracy alongside observation tokens, reasoning tokens, cost, artifact coverage, and deduplicated evidence intake, allowing them to verify that performance gains correspond to genuinely new evidence being read rather than just more prompting.

Why This Matters

Impact on research: The paper challenges the assumption that better RCA requires bigger models or more reasoning compute. It reframes attribution as an exploration problem, showing that search strategy — not raw capability — is often the bottleneck. It also introduces a much larger benchmark (MegaRCA-Mix) that exposes how poorly frontier models perform on realistic, execution-heavy logs, where F1 scores remain below 0.65.

Real-world applications:

• Production agent debugging: Automatically triaging failures in deployed agents, routing the fault to the model, harness, environment, or grader so the right team fixes it.
• Security forensics: Reconstructing long intrusion timelines, as in the cited Hugging Face incident where investigators had to review 70,000+ agent messages to recover the attack sequence.
• Benchmark maintenance: Flagging broken tasks, mistranslated instructions, service failures, and stale environments so that evaluation suites stay trustworthy.
• Post-training data curation: Turning failed trajectories into actionable feedback that can inform fine-tuning objectives or automated agent self-improvement loops.

Industry relevance: As agents shift from short demos to long-horizon, sandbox-based deployments, execution logs are growing faster than human review can scale. Reliable automated attribution is a prerequisite for agent observability tooling, safety auditing, and continuous self-improvement pipelines — making this directly relevant to any organization running or evaluating agents in production.

Future Directions

• Handling unobserved reasoning. Many proprietary models hide their thinking tokens, so the true causal step may be invisible. Extending the approach to partially observable traces, or working with providers to expose internal reasoning, is a natural next step.

• Adaptive stopping criteria. Continual Search helps when evidence remains unread but hurts when the trace is small. A mechanism that detects evidence exhaustion and stops iterating — instead of applying fixed turns — could eliminate the regressions observed on AgentRx and Who&When.

• Cleaner failure taxonomies and ground truth. The authors flag possible category overlap and label noise in human annotations of massive logs. Better annotation protocols and more orthogonal taxonomies would raise achievable ceiling scores.

• Extending to autonomous repair. The paper hints at agentic self-improvement. A logical extension is closing the loop: using attribution output to automatically propose harness fixes, environment repairs, or training-data updates rather than only producing diagnoses.

Target Audience

This paper is most valuable to agent reliability and evaluation engineers, LLM benchmark maintainers, and AI safety/forensics practitioners who need to diagnose failures in long-running autonomous systems. It also benefits applied ML researchers working on inference-time compute, agentic judges, or test-time search strategies, and product teams deploying agents in production who need automated observability. A basic understanding of agent trajectories and evaluation metrics suffices; no deep theoretical background is required.

Authors’ abstract

The increasing deployment of AI agents in long-horizon tasks yields massive execution logs. Diagnosing failures within these records is crucial for reliability, as it transforms outcome-level signals into actionable interventions. The sheer scale of the data renders human review impractical, driving the need for automated root-cause attribution (RCA). However, automated RCA methods using LLMs suffer from low diagnostic accuracy, especially as execution traces grow larger. They struggle because relevant information is often sparse, distributed across distant actions, and disconnected from the visible failure, reducing root-cause attribution to a massive search problem. Existing RCA methods typically rely on one-shot LLM judgments to diagnose failures from execution traces. While effective for shorter trajectories, these judges tend to settle on a plausible diagnosis early, leaving critical evidence in longer traces unexamined. We introduce Continual Search, an iterative framework that nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence. We evaluate Continual Search across four existing RCA benchmarks. Recognizing the lack of massive execution traces in current benchmarks, we introduce MegaRCA-Mix to evaluate RCA at scale. MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks. Across multiple benchmark suites and model families, Continual Search consistently improves attribution performance. On MegaRCA-Mix, for example, it improves GPT-5.5's F1 score by more than 40\%, from $0.349$ to $0.498$. More interestingly, within the same model family, lower-tier models can even surpass their higher-tier counterparts, demonstrating that effective search supersedes raw model scale.