The Pulse
Together AI Details GPU Node Auto-Repair With Per-Fault Controls
Together AI says its GPU cluster auto-repair system is generally available, with health checks, repair recommendations and customer-set approval policies. The system escalates from reboot to reprovision or migration and sends vacated machin
AI.info Team · Written with AI · fact-checked ·
“The hard part is everything between the fault and a healthy node again.”
Pavneet Ahluwalia, Dongsheng Zhang and Barbara Kazanowska
Together AI published that assessment on October 27, 2026, as it detailed how its automatic repair system handles failures in Together GPU Clusters. The company says passive health checks, repair recommendations, customer remediation policies, workload-idle controls, safety measures, and repair through the console and API are generally available. The system is designed to address faults that can stall distributed training or take down inference replicas, including NVIDIA Xid errors, double-bit ECC errors, GPUs disappearing from the PCIe bus and unstable InfiniBand links.
Why a NotReady node is not enough
Together’s system does not treat a Kubernetes node marked NotReady as sufficient grounds for repair. A node can become unreachable briefly without being broken, and a storage or network event can make many nodes fail the same readiness check at once. The company says each repair must be backed by a failed health check and supporting evidence.
Detection runs both inside customer virtual machines and on the underlying host. GPU exporters inside the VM inspect kernel logs and DCGM data for problems such as Xid errors, PCIe loss, ECC faults, row-remap failures, PCIe replay rates and thermal throttling; Node Problem Detector watches for filesystem, kernel and service failures. On Slurm clusters, an exporter reports nodes marked DOWN or DRAIN, while host-side checks monitor InfiniBand links and host reachability.
Alerts carry a plain-language description and machine-readable evidence. Together says the evidence can include an Xid code and its NVIDIA mnemonic, or the peak temperature and throttle count for thermal throttling. The same alert information is exposed in the console, API and customer notifications.
Customers set the rules for unattended repairs
Together chooses the recommended repair for each fault, aiming to use the lightest action that addresses it. Customers decide whether recommendations wait for approval or run automatically, and can set consent by fault and node. The company gives automatic approval for reboots tied to Xid codes 119 and 95, while keeping migrations behind approval, as one possible configuration rather than a universal default.
Customers also control how long a repair waits for work to finish. The default maximum wait is two hours, with a 24-hour cap; a separate option to avoid interrupting running jobs removes the time limit. On Kubernetes, the system waits until no running pod on the node requests a GPU; on Slurm, it checks whether the scheduler reports an allocation.
For teams with their own detectors, repair requests can also come through the API or console. Node-label remediation contracts, which let a tool such as NVSentinel or a custom controller request a repair, remain in preview. Together says those requests still pass the system’s evidence checks, capacity checks and repair limits.
Reboots escalate; migrations trigger a hardware test
The repair sequence is to cordon the node, wait according to the customer’s policy, drain it, perform the repair and return it to service. Available actions range from rebooting the virtual machine, which preserves local scratch and temporary data, to rebuilding it on a healthy host or migrating it to different hardware. Removing a node for return-to-manufacturer service remains a human action.
Repeated faults move the recommendation up an escalation ladder: after the same action has been tried twice for a recurring fault, the next recommendation advances from reboot to reprovision, or from reprovision to migration. Attempts older than a week do not count. Migration also requires evidence of a critical alert on the node, either currently firing or recorded on the same machine within the past ten days; without that evidence, the request becomes a quick reprovision.
After a migration or removal, Together automatically runs the vacated physical machine through validation before returning it to the pool. A test virtual machine runs GPU, NCCL, InfiniBand and storage checks. A pass returns the machine to service, an explicit failure creates a hardware ticket and keeps it out of the pool, and an aborted test restores the machine’s prior state without condemning it.
Safeguards target correlated failures
Together’s controls are designed to limit repairs when many alerts share a cause. Signals must persist for a configured duration before generating recommendations; a repaired node has a cooldown of roughly 30 minutes, and the operator runs one repair per node pool at a time. A circuit breaker can stop a burst of correlated automatic repairs, since replacing nodes one by one would not fix a cluster-wide storage or network problem.
The post describes how the system makes decisions but gives no measured recovery times, downtime reductions or pricing. Its central operational choice is to separate detection from permission: Together identifies the fault and recommends the action, while customers decide which failures may trigger repair without an approval and how long the platform should wait before acting.