Loom: Weaving Diagnostic Strands into Free-Text Consensus via Embedding-Space Reweighting

AI summary

Overview

Research area: Applied NLP / AIOps, specifically automated Root Cause Analysis (RCA) for large-scale computing infrastructure, generative consensus aggregation, and weak supervision extended to free text.

Technical level: Advanced. The paper combines an iterative embedding-space algorithm with a production deployment report and benchmark evaluation, and assumes familiarity with embeddings, weak supervision, and LLM agents.

Scope: Loom is a deployed generative consensus framework that aggregates open-form, templated diagnostic hypotheses from programmatic heuristics by projecting them into a continuous embedding space and resolving conflicts via iterative centroid-based reweighting before a single LLM synthesis call.

Authors: Ron Begleiter, Katya Egert Berg, Gilad Saban, Gil Shabat (NVIDIA; Tel Aviv, Israel). arXiv:2609.02649v1 [cs.AI], 02 Sep 2026.

What This Paper Is About

Diagnosing failures in large computing environments requires turning noisy, conflicting textual evidence into a single reliable explanation. Monolithic LLM agents can express anything but suffer from context limits, compounding hallucinations, and prohibitive latency, while traditional weak supervision is mathematically restricted to discrete predefined classes and cannot aggregate free-text hypotheses. Loom's goal is to aggregate unconstrained, episode-specific textual hypotheses in a continuous embedding space so that a single lightweight LLM call can produce a coherent, auditable root cause narrative.

Key Contributions

1. A generative consensus framework deployed for real-world RCA that extends weak supervision from discrete labels to templated, episode-specific hypotheses emitted by modular programmatic heuristics called Diagnostic Strands (DSs).
2. An iterative centroid-based reweighting algorithm that resolves conflicting signals deterministically in embedding space, bypassing costly LLM debate loops.
3. Real-world case studies of Loom deployed in NVIDIA production datacenters, including a distributed training job failure and a silent performance degradation without hard failures.
4. An OpenRCA evaluation placing Loom on the accuracy–efficiency Pareto frontier, with a reported ~26–33× inference speedup over the RCA-Agent baseline on every dataset, plus a discussion of negative results (static redundancy detection) and lessons learned.

Main Findings

• Pareto frontier on accuracy vs. efficiency: Loom uses 1 LLM call and ~22 seconds per incident on all four OpenRCA dataset instances, versus RCA-Agent's ~62 LLM calls and ~567 seconds per incident (~26× speedup).
• Matches the agent on two datasets: On Bank, Loom + Claude 4.6 reaches 38.97% strict and 51.22% partial accuracy versus the agent's 40.44% strict and 49.15% partial — Loom exceeds partial accuracy there. On Market-2 both reach 35.90% strict (Loom 50.85% partial vs. agent 50.76%).
• Trails on two datasets: On Market-1, Loom reaches 28.57% strict versus the agent's 40.00% (an 11.43 pp gap); on Telecom, 29.41% strict versus 41.18% (an 11.77 pp gap).
• Oracle scores show the consensus often finds the answer: Oracle strict accuracy on the Loom slate is 70.51% for Market-2 and 70.00% for Market-1. On Telecom the Oracle itself is 35.29%, below the agent's 41.18%, indicating a DS catalog coverage bottleneck rather than a synthesis bottleneck.
• Bank performance at query-difficulty extremes: Easy strict 46.77% (Loom + Claude 4.6) versus 41.94% (agent); hard strict 41.18% versus 29.41%. The agent leads on middle-difficulty queries (42.11% versus 29.82%).
• Consensus decouples performance from LLM scale: Replacing Claude 4.6 with Llama-3.1-8B drops overall Bank strict accuracy by only ~3.7 pp (38.97% to 35.29%); on easy queries the two judges tie at 46.77%. The 8B synthesizer yields a ~33× speedup and ~17 s average time per item versus 566.8 s for the agent.
• Iterative reweighting helps: Removing it costs 5.88 pp strict accuracy on the 136-incident Bank benchmark (38.97% to 33.09%).
• Negative result on static redundancy detection: Removing the DS-docstring redundancy step improves strict accuracy by the same 5.88 pp (to 44.85%), because docstring-level grouping is too coarse for a small expert-curated catalog and compresses whole reason-families out of the top-K list.
• Judge Overfocus: On Telecom, when faced with multiple high-confidence candidates with saturating anomaly scores across different nodes, the single-shot LLM often commits to an incorrect but highly anomalous fault type; on Market-1 it exhibits "same-family reason confusion."
• Deployment case studies: Loom diagnosed a failure across a 512-node cluster, naming node-011-T07 and node-011-T15 with 480M link retries and a raw bit error rate of 7.00 × 10⁻⁶, while describing 337 downstream hosts with throughput drops and a 507-host blast radius. In a silent degradation case with a 30% throughput drop, DSs received weights of 0.94 (ds_broken_dcqcn_loop), 0.85 (ds_ecn_marked_high, ECN marks > 50 k/s), 0.82 (ds_cnp_handled_low, < 10/s), and 0.76 (ds_rx_pause_high).

Methodology in Plain English

Instead of asking one large model to investigate everything, Loom splits troubleshooting knowledge into small, purpose-built programs called Diagnostic Strands. Each strand looks at structured data — logs, counters, telemetry — and either produces a filled-in sentence template naming a suspected cause with specific hostnames, metrics, and timestamps, or abstains. These strands are authored by domain experts or extracted offline by LLM agents from historical incident tickets and documentation.

At inference time the whole catalog is run against a single incident. Fire outputs are converted to vector embeddings. The algorithm then alternates two steps: computing a weighted centroid of all embeddings, and re-assigning each strand's weight to its cosine similarity with that centroid. Redundancy is handled by dividing weights within groups of similar strands (based on docstring similarity against a threshold τ). This converges in milliseconds and yields a ranked, denoised list of hypotheses. A single LLM then synthesizes one narrative from that ranked list, strictly prompted to preserve technical details verbatim, prioritize higher-weighted observations, and not invent facts or reference internal function names or weights. The pipeline is deterministic at temperature 0.

Evaluation used the OpenRCA benchmark across three datasets reported as four instances (Bank, Telecom, Market-1, Market-2), with structured JSON output matching the benchmark schema. Ablations on Bank isolated iterative reweighting and redundancy detection, and an Oracle analysis checked whether the correct hypothesis appeared in the consensus slate.

Why This Matters

Impact on research: The paper challenges the assumption that conflict resolution requires iterative LLM deliberation. It shows that moving aggregation into embedding space can match agentic accuracy on some datasets at a fraction of the cost, and it provides a candid negative result about static redundancy detection that other weak-supervision-style systems can act on.

Real-world applications:

• Root cause analysis for distributed training jobs and GPU clusters, where node failures are masked by collateral damage across hundreds of hosts.
• Silent network performance degradation detection, such as diagnosing a broken DCQCN congestion-control feedback loop from switch-side ECN markings and host-side CNP counters.
• Automated log and telemetry triage in AIOps pipelines that currently rely on human Subject Matter Experts.
• Air-gapped or cost-sensitive environments that cannot call large hosted models and need an 8B-parameter local synthesizer.

Industry relevance: The framework targets the practical constraints that block LLM deployment in operations: latency, token cost, hallucination risk, and auditability. Its deterministic, geometry-based aggregation gives operators an inspectable evidence trail and bounded coverage — they know exactly which failures the DS catalog can address. Loom was deployed in NVIDIA production datacenters to diagnose both hard distributed job failures and silent networking degradations.

Future Directions

• Hybrid pipelines that use Loom to rapidly surface candidate root causes and a lightweight agent to perform final disambiguation, which the authors suggest could close the accuracy gap on Market-1 and Telecom while preserving efficiency.
• Dynamic redundancy handling conditioned on outputs rather than statically bound to docstrings, since static grouping hurt accuracy on the curated Bank catalog.
• Expanding DS catalog coverage on datasets like Telecom, where the Oracle score of 35.29% indicates the bottleneck is partially in catalog coverage rather than synthesis.
• Reducing the cold-start requirement for novel "Black Swan" failures, which currently needs a human expert or offline extraction pipeline to author and validate a new Diagnostic Strand before Loom can diagnose the issue.
• Improving single-shot synthesis to resist Judge Overfocus and same-family reason confusion, given that the correct hypothesis was present over 70% of the time in oracle evaluations.

Target Audience

Engineers and researchers building production AIOps, observability, or automated diagnosis systems; practitioners working on weak supervision and programmatic labeling who need to handle free text rather than discrete classes; and applied ML teams evaluating the cost–accuracy trade-off between agentic LLM loops and deterministic aggregation. It is also useful for infrastructure operators and Subject Matter Experts who need to understand what an automated RCA system can and cannot diagnose.

Authors’ abstract

Aggregating noisy, conflicting textual hypotheses into a reliable consensus is a fundamental challenge when deploying NLP systems in real-world industrial settings. While monolithic Large Language Model (LLM) agents offer unbounded expressivity for tasks like Root Cause Analysis (RCA), they suffer from context limits, compounding hallucinations, and prohibitive inference latency. Traditional weak supervision offers statistical rigor but is mathematically restricted to discrete classes. We present Loom, a generative consensus framework deployed for real-world RCA that bridges these paradigms. Loom aggregates open-form hypotheses emitted by modular heuristics (diagnostic templates dynamically populated with episode-specific entities, times, and metrics) by projecting them into a continuous embedding space, and resolves conflicting signals with an iterative centroid-based reweighting algorithm. The resulting consensus weights ground a single lightweight LLM synthesis step. Evaluated on the OpenRCA benchmark, Loom occupies the accuracy--efficiency Pareto frontier: it matches a state-of-the-art autonomous agent on Bank and Market-2 and trails on Market-1 and Telecom, while using a single LLM call per incident on all four datasets ($\sim$26$\times$ faster; $\sim$33$\times$ with an 8B-parameter synthesizer). We discuss our deployment experience, highlighting lessons learned regarding the trade-offs between agentic depth and inference latency, negative results in redundancy detection, and how deterministic consensus fosters trust among Subject Matter Experts~(SMEs).