The Pulse
NVIDIA Opens Its Distributed AI Failure-Attribution Tool
NVIDIA has open-sourced Attribution, a tool that correlates workload logs, metrics and scheduler state to diagnose distributed AI failures. The system returns a structured result with a suspected root cause, affected nodes, supporting sympt
AI.info Team · Written with AI · fact-checked ·
NVIDIA published its open-source workload-attribution system on Nov. 2, 2026. Attribution is designed to identify where a distributed workload failure began, rather than treating every error reported across a cluster as an independent fault. GPU and network-fabric events can trigger errors on other ranks, scheduler changes and a flood of messages; the exception that stops a job may not identify the initiating problem.
One diagnosis from scattered workload evidence
The tool correlates workload-scoped logs, metrics, telemetry and scheduler state, then returns a structured result with an outcome, root cause, affected nodes, supporting symptoms and an evidence trail. NVIDIA says Attribution has run in production across its AI infrastructure for years, analyzing millions of workloads each week across hundreds of thousands of GPUs. In one large RDMA-backed cluster, the system correlated port errors, connection resets and scheduler events to identify patterns around specific host and switch ports, helping teams target fabric repairs and congestion tuning.
Attribution has two deployable components. The Attribution Service exposes the analysis engine through an HTTP API, while the Python-based Attribution Blueprint connects cluster schedulers and telemetry sources to the Service, enriches its result and writes it to an output destination. The separation lets teams use the same attribution logic while adapting integrations to their own clusters.
How the four-stage process traces a failure
The Blueprint first identifies a workload attempt using scheduler identifiers or an explicit execution window and node allocation. Its collectors then gather workload-specific observations from application logs, system logs, GPU driver and Fabric Manager messages, scheduler history, and node and network metrics. The Service extracts events—including GPU XIDs, NCCL failures, scheduler transitions and metric conditions—and applies deterministic rules to distinguish a cause from symptoms that followed it.
The Blueprint returns the verdict with the workload’s terminal state, attributed domain, symptoms, affected nodes, and a record of collector coverage and errors. In NVIDIA’s example, a multi-rank training workload develops an XID 149 fault on one GPU, followed by NCCL errors on other ranks. Attribution identifies an NVLink signal-integrity issue on node-02 as the cause, retains the NCCL reports as supporting symptoms and classifies the job as a hardware failure.
Connectors cover common cluster data sources
The Blueprint ships with execution-context connectors for Slurm, Kueue and static inputs, alongside log and metric connectors for Loki, local files, Kubernetes pod logs and PromQL. Output options include stdout and SQLite. Teams can place Blueprints near cluster data sources and use either a dedicated Attribution Service for each Blueprint or a shared service for multiple Blueprints.
NVIDIA’s walkthrough runs the Service and a local dashboard in Docker, then uses the Blueprint to process sample workload evidence included in the repository. The sample lets users inspect diagnoses, affected nodes, symptoms and evidence before connecting a live cluster. NVIDIA says teams can add rules for new failure modes, build connectors for their own schedulers and telemetry, and route results into incident, recovery and fleet-management systems.
Incomplete evidence can leave the cause open
NVIDIA cautions that attribution quality depends on the quality and completeness of telemetry. When multiple causes fit the evidence, the result marks the attributed domain as ambiguous rather than converting incomplete evidence into a definitive diagnosis. The rule catalog also has a deployment constraint: rules load at Service startup, and changes must be committed and shipped with the Service rather than uploaded dynamically through its HTTP API.