Monitoring Training Runs and Making Stop/Continue Decisions

At hour fourteen, validation loss rises while training loss keeps falling. The GPUs stay busy, but throughput has dropped and gradient norms oscillate after a data refresh. Wait, restart, or terminate?

No single chart answers that. Useful monitoring ties several signals to the current training phase, the known failure modes, the remaining budget, and a decision policy written down in advance. The rest of this lesson works from runs whose operators published their telemetry and the decisions they took on it: PaLM's twenty loss spikes, OPT-175B's restart chronicle, the 466 interruptions Meta counted in a 54-day window of Llama 3 405B pre-training, and the alarm statistics hospitals and federal investigators have collected on what happens to a monitoring channel nobody trusts.

These patterns look familiar, and each needs context before action. The cost of skipping that context has been measured.

An early-warning score in daily hospital use scored far worse when somebody else checked it. Epic's proprietary Sepsis Model was validated externally at Michigan Medicine across 27,697 patients and 38,455 hospitalizations. Wong and colleagues reported a hospitalization-level AUC of 0.63 (95% CI, 0.62–0.64) in JAMA Internal Medicine in 2021. The vendor had reported 0.76–0.83. The headline number was not the deployed number, and the operational consequence was counted rather than described: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.”

A second, independent validation covered 145,885 emergency-department encounters at two county EDs. It measured a sensitivity of 14.7% within a six-hour window. A monitor that alerts on 18% of admissions and misses two-thirds of the cases it exists to catch is a dashboard reading, not a verdict.

A healthy top-line metric can hide failure lower in the stack. The bottom layer, resource and system health, is the one whose failure rate has actually been published.

In a 54-day window of Llama 3 405B pre-training, on up to 16,384 H100 GPUs, Meta's team counted every interruption: “During a 54-day snapshot period of pre-training, we experienced a total of 466 job interruptions. Of these, 47 were planned interruptions due to automated maintenance operations such as firmware upgrades or operator-initiated operations like configuration or dataset updates. The remaining 419 were unexpected interruptions, which are classified in Table 5.” Approximately 78% of the unexpected interruptions were attributed to confirmed or suspected hardware issues. GPU issues alone accounted for 58.7%; faulty GPUs, 148 events (30.1%); GPU HBM3 memory, 72 events (17.2%). The run still achieved higher than 90% effective training time.

That is a hardware interruption roughly every three hours, and not one of them is visible in a loss curve. Nor is it one company's bad luck. ByteDance's MegaScale team reports a production run of a model with hundreds of billions of parameters, on more than 10,000 GPUs, lasting several weeks. Training was restarted over 100 times.

Read the five layers with that base rate underneath them. Resource and system health carries throughput, memory, communication, I/O, checkpoint time and hardware errors. Data delivery carries batch composition, freshness, augmentation rates, masks, labels and duplicate or corrupt records. Numerical health carries finite losses, gradient scale, overflow, underflow, clipping and parameter statistics. Optimization behavior carries training loss, update-to-weight ratios, learning-rate phase and gradient noise. Generalization evidence carries validation metrics, slices, calibration and behavior on reserved examples. The upper four are all being sampled on top of a substrate that, on two published clusters, fails on a schedule of its own.

The action should follow from diagnosis and recoverability, not from impatience.

Continue when signals remain inside expected phase-specific ranges: loss follows the anticipated schedule, validation evidence has not invalidated the run, resources remain within budget, and no data or numerical contract is broken.

Intervene when the run may be recoverable without invalidating its evidence — a transient dataloader slowdown, for instance. Inspect before changing semantics. Preserve a decision log. Remember that ad hoc changes create a new experiment. PaLM's spike handling is the intervention done properly: a restart “from a checkpoint roughly 100 steps before the spike started, and skipped roughly 200–500 data batches”, and then an ablation to establish what had actually gone wrong. The restart was a diagnosis rather than a hope.

Terminate when continuing cannot produce trustworthy or worthwhile evidence: leakage or corrupt targets, persistent non-finite state, or an exhausted budget with no credible path. The first action on termination is to preserve artifacts before shutdown. The OPT and Llama 3 numbers below exist only because somebody kept the record.

Record the observation, the competing explanations, the action, the owner, the timestamp, and the evidence expected next. That makes a later postmortem more informative than a folder of charts. Across projects it also exposes repeated patterns: teams that routinely ignore clipping, stop too late, or compare runs after changing several variables at once.

The OPT-175B team published its own notes, which is rarer than it sounds. The paper reports that “hardware failures contributed to at least 35 manual restarts and the cycling of over 100 hosts over the course of 2 months”, alongside an estimated “70+ automatic restarts due to hardware failures”. The team's public chronicle, posted on 7 January 2022, puts the total higher still: “These notes cover ~90 restarts over the course of training the lineage of this current model”. The same document records the background rate in a parenthesis, “~2 machines go down every day”.

Thirty-five manual restarts in the paper, about ninety in the logbook. The two figures cover different scopes, and only the existence of the logbook makes that distinction visible at all. None of it appears in a loss curve. It survives because somebody wrote it down while it was happening.

Static thresholds often fire during normal phase changes — warmup, evaluation, checkpointing, batch-length variation. Repeated false alarms teach operators to ignore the system. Hospitals have measured that base rate, and then measured what ignoring it costs. The Joint Commission's Sentinel Event Alert 50, dated 8 April 2013, states: “It is estimated that between 85 and 99 percent of alarm signals do not require clinical intervention, such as when alarm conditions are set too tight; default settings are not adjusted for the individual patient or for the patient population; ECG electrodes have dried out; or sensors are mispositioned.” A direct measurement agrees with the estimate. Drew and colleagues logged 2,558,760 unique alarms across 77 ICU beds and 461 patients in 31 days, and found that “88.8% of the 12,671 annotated arrhythmia alarms were false positives.”

The consequence is recorded in the same alert. Between January 2009 and June 2012 the Joint Commission's Sentinel Event database recorded 98 alarm-related events. Eighty resulted in death, 13 in permanent loss of function and five in extended care. Among the contributing factors it counts “Alarm signals inappropriately turned off (36)”. Muting is the rational response of an operator facing a channel that is wrong most of the time. It is also how the channel kills someone.

Every alert should therefore name its scope, persistence rule, owner, diagnostic link, and permitted response. A phase-blind threshold on a training run is the same instrument. It will be right occasionally, wrong constantly, and eventually switched off by the person it was built to protect.

A review should be short enough to repeat and rigorous enough to change a decision. Six steps.

First, confirm identity: run ID, code, data snapshot, configuration, and current phase. Second, check hard invariants: finite values, data contracts, masks, synchronization, and checkpoint integrity. Third, compare with expectation using baseline bands and prior runs rather than an isolated visual impression — PaLM's ablation is what this looks like done properly, the suspect batches replayed from an earlier checkpoint to find out whether the data or the pair was to blame. Fourth, inspect the slices linked to the intended use rather than the aggregate. Overall accuracy of 87.9%–93.7% was compatible with a 34.7% error rate on one subgroup, and an AUC of 0.63 was compatible with alerting on 18% of hospitalizations while missing 67% of the cases. Fifth, estimate remaining value, comparing plausible improvement with time, money, and opportunity cost. Sixth, record the decision — continue, intervene, or terminate — with its evidence and the next review trigger.

Non-finite values, missing batches, corrupt checkpoints and hard budget limits are strong candidates for automatic handling. Ambiguous validation changes or rare-slice tradeoffs usually need a human review. Two published frontier-training operations drew that boundary in almost the same place and, unusually, measured where it fell. MegaScale reports: “With the robust training framework, over 90% of software and hardware faults are automatically identified and fixed by the techniques detailed in §4. The rest of the problems are handled with the help of the troubleshooting tools described in §5.” On the Llama 3 side the human residue is smaller still. Across the 419 unexpected interruptions in those 54 days, “Despite the large number of failures, significant manual intervention was required only three times during this period, with the rest of issues handled by automation.”

Notice what was automated in both cases. It is detection, isolation and restart — MegaScale's robust training framework with heartbeat-based anomaly detection and diagnostic tests, Meta's automated handling of interruptions. It is not the judgment about whether the run remains worth finishing. Three human decisions across 419 unexpected interruptions is the target ratio, and it is reachable only because the rest were invariant violations with a defined response.

Automation should also fail safely and preserve state. An automatic stop that deletes evidence, or restarts with changed semantics, can make diagnosis harder than the original incident. It also destroys exactly the record that let the Llama 3 and OPT teams publish their numbers at all.