RAFT: A Stateful Retrieval-Augmented Framework for Troubleshooting Agents

AI summary

Overview

Research area: Retrieval-augmented generation (RAG) and knowledge-graph retrieval for enterprise customer support and IT troubleshooting agents.

Technical level: Intermediate — readers should know what RAG, embeddings, and GraphRAG are, but the core ideas are explained in accessible terms and the evaluation is retrieval-only rather than a full agent pipeline.

Scope: The paper proposes RAFT, a stateful retrieval framework that represents each resolved support case as a chain of timeline entries, retrieves at the entry level rather than the document level, and demonstrates gains over vanilla RAG and two GraphRAG baselines on a synthetic Windows Server benchmark plus a real Apache Jira evaluation set.

What This Paper Is About

Troubleshooting a support ticket is a multi-stage investigation: a symptom appears, hypotheses get tested, evidence accumulates, and a root cause is confirmed. Existing RAG systems treat old support cases as flat documents, so they retrieve noisy, context-free chunks that ignore where an investigation currently stands. RAFT reframes each closed case as a directed sequence of intermediate troubleshooting states, allowing retrieval to match the current stage of an active case against the corresponding stage of a past one.

Key Contributions

1. A stateful RAG architecture that abstracts each closed historical case into a directed chain of timeline entries, retrieves over those evolving intermediate states, and returns the full parent-case trajectory anchored at the matched entry.
2. An optional case-level graph that links cases through a configurable similarity view (in experiments, concatenated root cause and resolution text scored by a hybrid of semantic and BM25 similarity fused via Reciprocal Rank Fusion), enabling expansion to complementary cases that do not match at the entry level.
3. A public synthetic development benchmark of 826 Windows Server support cases built from Microsoft Learn documentation, with a reproducible protocol that probes retrieval at multiple progress points of an active case.
4. An open, reproducible evaluation of the retrieval layer alone — avoiding dependence on production deployment — including an Apache Jira transfer set, with code, benchmark, and evaluation data released.

Main Findings

• RAFT wins on every metric at every stage. On the synthetic benchmark, RAFT reaches 84.2% Case Hit at 0% progress (symptom only), rising to 88.8% at 60% progress, compared with 67.3% and 76.9% for vanilla RAG, 65.0% and 71.1% for HippoRAG2, and 42.1% and 58.3% for Fast-GraphRAG.

• Gains are statistically significant. Against vanilla RAG — the strongest baseline — the Case Hit improvements hold at all three progress points under bootstrap resampling clustered by root-cause group.

• GraphRAG baselines underperform vanilla RAG here. Fast-GraphRAG is substantially worse at all progress levels, and HippoRAG2 fails to surpass vanilla RAG, consistent with prior findings that entity-centric graph construction offers little benefit when the task does not require cross-document hierarchical reasoning.

• Retrieval matches the right stage, not just the right case. The matched timeline entry moves steadily deeper into the historical case as the query carries more context: 9.1% depth at 0% progress, 20.0% at 30%, and 54.0% at 60%. This confirms the entry-level indexing is behaving as designed.

• Coverage metrics improve in parallel. Root-cause coverage rises from 64.9% to 68.9% and resolution-steps coverage from 56.3% to 60.5%, so RAFT retrieves not just a matching case but more of the diagnostic and remediation evidence needed to act on it.

• The advantage transfers to real data. On 30 human-audited Apache Jira duplicate groups with 570 distractor issues, RAFT outperforms vanilla RAG by +16.7, +17.3, and +10.5 percentage points at 0%, 30%, and 60% progress, using no modification to extraction prompts, schema, models, or retrieval procedure.

• RAFT is more robust to noisy queries. Under perturbation with off-topic content, typos, and dropout, RAFT degrades less than vanilla RAG.

Methodology in Plain English

Each closed case is processed independently by a worker–reviewer workflow. A worker reads the case history in bounded batches, carrying an evolving structured state forward across passes, and can query the raw source records on demand. The state is updated through JSON Patch operations, so later evidence can correct earlier interpretations without regenerating everything. A reviewer then inspects the finished state and revision history, fixes inconsistencies, and produces a structured assessment — which can include actionability labels used for filtering, addressing the problem that some tickets close for administrative reasons without documenting any useful investigation.

The result is a compact timeline of entries, where each entry marks a meaningful state transition (a new hypothesis, a confirmed root cause, a proposed fix) rather than a fixed-size text chunk. Every timeline entry across every indexed case is embedded. At query time, entries are ranked by the same hybrid semantic-plus-lexical score, and top entries are greedily promoted to their parent cases until a case count or token budget is reached. The agent receives both the full case and the anchor entry that triggered the match — so it can see which investigative state was matched, not just which ticket.

Evaluation deliberately targets the retrieval layer rather than a deployed agent, which removes the need for production access and makes the comparison reproducible. Queries are built from prefixes of held-out test cases at 0%, 30%, and 60% of turns, simulating successively deeper investigation. Three metrics are used: Case Hit (did retrieval surface a case sharing the same root cause and resolution), Root Cause Coverage (fraction of gold root-cause claims entailed by retrieved context, judged by an LLM), and Resolution Steps Coverage (the same for the gold remediation procedure).

Why This Matters

Impact on research. Public multi-stage troubleshooting corpora with labeled similar cases are essentially nonexistent, and prior work in this space — including the closest comparable system by Xu et al. (2024) — uses private data and production metrics that make comparison impossible. RAFT contributes a reproducible benchmark, a released evaluation set, and a retrieval-layer evaluation protocol that others can extend without deploying anything. It also supplies a counterpoint in the ongoing RAG-versus-GraphRAG debate: when the task is surfacing similar cases rather than reasoning across documents, a well-structured sequential representation outperforms entity-graph pipelines.

Real-world applications:

• Enterprise IT helpdesks routing and resolving Windows Server, Active Directory, or networking tickets by matching the current investigation state to past resolved cases.
• Software support organizations, where Jira-style issue trackers already contain duplicate linkage and rich contributor histories.
• Field-service and operations teams whose work unfolds in stages (initial alert, diagnosis, remediation) and where the appropriate precedent depends on how far the investigation has progressed.
• Any RAG deployment over long, noisy, multi-turn historical records where naive chunking wastes context budget and fragmenting cases across chunks prevents coherent reasoning.

Industry relevance. The paper comes from Microsoft and is accepted to the EMNLP 2026 Industry Track, which frames it explicitly as a production-motivated contribution. The architecture addresses practical enterprise constraints that academic RAG work often ignores: per-call token caps on retrieval tools, the need to filter non-actionable closed tickets, and personally identifiable information that must be abstracted before indexing. The emphasis on evaluating a component instead of a whole deployed system is a pragmatic response to how difficult reproducibility is in this domain.

Future Directions

• Scale and realism of evaluation. The main quantitative results come from a synthetic benchmark of moderate scale with cases averaging roughly 2,800 tokens. Production corpora contain far more cases with much longer histories, and it is unclear how entry-level retrieval behaves at that scale and length.
• Strengthening the real-world evidence. The Jira evaluation covers only 30 audited duplicate groups and carries no confidence intervals, so the authors treat it as directional rather than conclusive. Larger, more diverse real datasets with proper uncertainty quantification would settle the transfer question.
• End-to-end agent outcomes. The work stops at retrieval and does not measure final diagnosis accuracy, resolution success, engineer productivity, or time-to-resolve. Whether better retrieval translates into better troubleshooting remains open.
• Indexing-model sensitivity and agentic query composition. The authors include an ablation on indexing-model capacity and a case study where the agent composes its own queries and filters, both of which suggest that how the agent interacts with the retrieval layer is itself a design lever worth exploring further.

Target Audience

Applied AI and ML engineers building RAG systems over enterprise support data; RAG and GraphRAG researchers interested in state-aware or case-structured retrieval; industry practitioners evaluating whether complex graph construction is worth the cost for their workloads; and technical leaders at software or IT service organizations designing agent-assisted helpdesk tooling who want a concrete, reproducible baseline before investing in a full deployment.

Authors’ abstract

Effective troubleshooting agents in enterprise customer support depend on retrieving actionable guidance from similar historical cases, yet existing retrieval-augmented generation (RAG) systems treat support cases as static documents and overlook their multi-stage, stateful nature. We introduce RAFT (Retrieval-Augmented Framework for Troubleshooting Agents), a stateful RAG framework that abstracts each closed historical case into a directed chain of timeline entries and retrieves at the entry level, surfacing cases whose intermediate states match the active case and returning the parent-case trajectory anchored at the matched state; an optional case-level graph links cases through a configurable similarity representation. We evaluate this retrieval layer directly, which, unlike evaluating a full agent system, requires no production deployment. Because public multi-stage troubleshooting data is extremely rare, we pair a synthetic benchmark built from Microsoft Learn Windows Server documentation with real Apache Jira issues carrying human-created duplicate labels. RAFT improves Case Hit over vanilla RAG and GraphRAG baselines at every stage of case progress, with statistically significant gains over the strongest baseline; the Jira results provide directional evidence that the advantage transfers to real case histories. We release our benchmark, implementation, and the Apache Jira evaluation set.