ORCA-bench: How Ready Are Language Model Agents for Oncall?

AI summary

Overview

• Research area: Benchmarking large language model (LLM) agents on production-style site reliability engineering (SRE), specifically oncall root cause analysis (RCA) over live telemetry and source code.
• Technical level: Advanced. Familiarity with microservice architectures, observability tooling (Prometheus, Jaeger, OpenSearch, Grafana, OpenTelemetry), and LLM agent evaluation is assumed.
• Scope: The paper introduces ORCA-bench, a benchmark that places general-purpose coding agents in a simulated oncall setting with a live OpenTelemetry-instrumented microservice system, and measures how well five frontier agents localize the root causes of user-facing incidents.

What This Paper Is About

Existing software-engineering benchmarks give an agent a frozen repository and a precise task specification (a bug report or a failing test) and grade it by re-running the test suite. Real oncall work is different: an engineer receives a vague complaint such as "users are having issues on the site," often hours after the incident began, and must reason over noisy metrics, logs, traces, and source code to produce a defensible diagnosis. ORCA-bench builds a production-fidelity environment for exactly this task and measures whether today's frontier coding agents are ready to perform it.

Key Contributions

1. The full oncall evidence stack in one environment. A live OpenTelemetry-instrumented microservice system (the Astronomy Shop) running six days of continuous simulated user load, with metrics in Prometheus, logs in OpenSearch, and traces in Jaeger queried through Grafana, plus full source-code access. The paper states this is the first SRE benchmark to expose all three artifacts an oncall engineer reaches for; the Shop consists of 19 microservices written in 13 different languages (including Go, Java, Python, Node.js, and C#).
2. A context ladder with jointly varied conditions. 1,079 RCA tasks generated by toggling real feature flags rather than a synthetic fault library, varying issue specificity (Easy/Medium/Hard), time-to-detection (TTD) across 15 min–24 hr, and co-occurring fault structure across five scenario types (isolated, independent, conflicting, cascading, and sequential), designed with expert SREs. Total raw telemetry is more than 50 GB over six days.
3. Symptom-level, hand-validated ground truth. Per-fault rubrics enumerating expected symptoms in metrics, logs, and traces, validated against telemetry by expert SREs; tasks are scored against the set of plausible root causes rather than a single label. A 40-task ORCA-bench Verified subset has every ground-truth label and every per-model score hand-confirmed.
4. A judge that humans re-scored from scratch. A GPT-5.4-based LLM-as-judge whose per-task scores were independently re-graded by hand on ORCA-bench Verified across all five evaluated agents, attaining a quadratic-weighted Cohen's κ_w = 0.90 against the human re-score.

Main Findings

• Frontier agents are far from being competent SREs. Across five frontier coding agents — Opus 4.7, Sonnet 4.6, GPT-5.5, GLM-5, and DeepSeek-V4-Pro — the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard. On the looser RCA depth metric over all 884 incident tasks, GPT-5.5 is highest at 48.8%, with Claude Opus 4.7 and Claude Sonnet 4.6 similar; the best RCA Accuracy over all 884 incident tasks is 30.6% (Claude Sonnet 4.6).
• Claude Fable 5 sets a higher bar on the verified subset. On the 32 incident tasks in ORCA-bench Verified, Claude Fable 5 achieves the highest RCA depth and RCA accuracy of 58.2 ± 5.8% and 40.6 ± 8.8%, compared to 49.2 ± 5.6% (21.9 ± 7.4%) for GPT-5.5 and 47.6 ± 5.7% (25.0 ± 7.8%) for Claude Opus 4.7.
• Hallucination is common. The weakest model, GLM-5, hallucinates an implausible root cause in 40% of incident reports; the range across models is 7% (DeepSeek-V4-Pro) to 40% (GLM-5).
• Reducing input context drops RCA accuracy by 19–50 percentage points across models. For hard tasks the agent sees only "users are reporting site issues," and the average number of ground-truth root causes rises to 4.41 ± 0.11, compared with 2.00 ± 0.06 for easy tasks. On Hard, the best model (Claude Opus 4.7) reaches only 10.0% RCA accuracy and 37.6% RCA depth; even on Easy tasks the best model (Claude Sonnet 4.6) manages only 58.7% RCA accuracy.
• Concurrent faults are a specific failure mode. On the Day 6 scenario with six co-occurring events, three models each identify exactly one root cause — and all different ones — missing the other five for a partial score of 0.5/3 = 0.167; GPT-5.5 finds four of six but misses two; DeepSeek-V4-Pro finds none. Manual scoring on ORCA-bench Verified found agents tend to lock onto louder symptoms and struggle to quantify severity increases and attribute them to distinct events.
• Removing source-code access hurts every metric. Without source code, RCA accuracy drops 9–16 percentage points and hallucination spikes for all models. Incident-time accuracy is relatively robust (Claude Opus 4.7: 82.6% to 79.9%), consistent with temporal information living in telemetry rather than code. Despite this, Claude Opus 4.7 and GLM-5 spend only 16% and 20% of commands reading source code, versus 72% and 70% on telemetry-related commands.
• Telemetry retrieval is inefficient and error-prone. 26–40% of telemetry calls either error out or return empty. GPT-5.5 is the most efficient in number of telemetry commands and has the highest success rate, with other models' failures largely driven by commands returning empty results. Using the collected ground-truth symptoms, GPT-5.5 also achieves the highest rate of capturing any ground-truth metric, log, or trace in its report.
• The reported gap is a lower bound. Because the testbed is a curated 50 GB / six-day system whose code and OpenTelemetry instrumentation are public, and tasks are investigated in isolation, the authors argue real production systems — larger, more dynamic, and more idiosyncratic — would be harder still.

Methodology in Plain English

The authors built the benchmark in six stages. First, they deployed the OpenTelemetry Astronomy Shop with Prometheus, OpenSearch, and Jaeger, exposing telemetry to agents through the Grafana API and source code through a terminal (raw telemetry was too large to hand over directly). Second, working with expert SREs, they composed a six-day schedule of 13 scenarios across five scenario types by firing built-in feature flags, structured around a "FAFO Friday" narrative: weekday buildup on Days 1–5 and a chaos peak on Day 6 (Friday) with six feature flags active concurrently.

Third, each task is parameterized by sampling an offset from incident start to report time, a TTD from {15 min, 1 h, 8 h, 24 h}, one of three report-time styles (exact, exact range, broad range), and a question issue specificity (easy, medium, hard). Fourth, GPT-5.4 rephrases the human-curated ground-truth symptoms at the sampled specificity. Fifth, ground-truth root causes are generated by enumerating candidate events active in the window from the start of the report day to the minimum of detection time and end of the report day, dropping candidates with no frontend symptoms, and having GPT-5.4 mark each survivor plausible or implausible against the user issue; the answer is the resulting set, and control tasks have the empty set. Sixth, symptoms are collected by firing each flag in a browser for frontend effects and by a semi-automated telemetry workflow validated by two expert SREs across 11 flags / 42 occurrences.

Evaluation runs in three stages: a detection check (did the agent correctly assert an incident, or its absence for control tasks), per-rubric grading on a 0–3 scale for each plausible root cause (0 = misaligned/completely missed, 1 = confirms symptoms only, 2 = correct progress beyond symptoms, 3 = nails the root cause and its supporting signals), and task-level aggregation into RCA accuracy, RCA depth, and hallucination rate. Agents are given a structured report template with four sections (Summary, Timeline, 5 Whys, Remediation) and are run with the Terminus-2 agent harness of Merrill et al. (2026), which provides an interactive tmux session and context compaction. A five-minute grace period after detection time accounts for telemetry still being ingested during an investigation.

Why This Matters

• Impact on research. Prior RCA benchmarks (AIOpsLab, ITBench, OpenRCA, SREGym) omit at least one of telemetry interface, raw live telemetry, or source code, and none systematically vary report specificity or time-to-detection. ORCA-bench supplies all of these together, along with human-verified symptom-level ground truth and a judge cross-checked against human re-scoring, giving the field a harder and more realistic target for agent evaluation.

Real-world applications:

• Production incident response: assessing whether coding agents can be trusted to help triage live outages before they are given production access.
• Observability and tooling design: the finding that 26–40% of telemetry calls error out or return empty points at agent-facing query ergonomics as a design problem, not only a model problem.
• Agent evaluation and procurement: the context ladder and the code-access ablation offer ready-made axes for comparing vendors' agents under realistic ambiguity.
• SRE training and triage support: the structured report template (Summary, Timeline, 5 Whys, Remediation) and the symptom rubrics are usable artifacts for reasoning about incidents.

Industry relevance. The headline result — best RCA Accuracy of 25.3% on Medium and 10.0% on Hard, with hallucination rates from 7% to 40% and uniform degradation when source code is removed — is a direct signal about readiness for AI-assisted reliability work. The authors position the measured gap as a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability.

Future Directions

1. Closing the action loop. The benchmark is deliberately read-only; a human SRE confirms a root cause by deploying a candidate fix and watching the symptom resolve. Adding mitigation and observing the feedback signal is described as a major engineering investment and a clear lever for better RCA itself.
2. Persistent memory and continual learning. Every ORCA-bench task is investigated cold. Giving agents memory across incidents, so they accumulate intuition about flaky services and noisy alerts, is a direction the authors suggest might eventually let agents outperform humans.
3. Scaling to realistic scale and dynamism. Extending beyond a 50 GB / six-day testbed to systems that generate terabytes per day on continuously evolving codebases, with drifting failure-mode distributions, would test agents against out-of-distribution faults.
4. Methods not studied. All models were evaluated with a single prompt template and a single agent harness; how much can be recovered via prompt engineering, structured workflows, or hybridization with causal-inference and graph-based RCA (e.g., Pham et al., 2025) is described as orthogonal and open.

Target Audience

Researchers and practitioners working on LLM agents for software engineering and reliability engineering; SRE and observability teams evaluating agent readiness for oncall; benchmark designers interested in human verification of ground truth and LLM-as-judge validation (the paper reports Spearman ρ = 0.92 and κ_w = 0.90 overall, with per-difficulty values of ρ = 0.91 / κ_w = 0.89 on Easy, ρ = 0.93 / κ_w = 0.96 on Medium, and ρ = 0.86 / κ_w = 0.84 on Hard); and model developers building tool-use agents for telemetry-heavy, long-horizon diagnostic tasks.

Authors’ abstract

Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access--with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's $κ_w=0.90$). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard--a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability. We release the public set at https://hub.harborframework.com/datasets/orca-bench/ORCA-bench.