Throughput, Failures, and Making a Long Run Finish

Device utilisation percentages are close to useless for training. A GPU spinning in a synchronisation barrier reports high utilisation while accomplishing nothing.

The metric that carries information is model FLOPs utilisation. Compute the floating point operations the model mathematically requires per token, multiply by tokens processed per second, and divide by the hardware's peak throughput. The result is the fraction of the machine you are converting into useful mathematics.

The word model in the name matters. It counts only the operations the model needs, not operations you performed. Activation recomputation runs an extra forward pass, and those operations do not count, correctly, because they buy memory rather than progress. A related metric, hardware FLOPs utilisation, does count them, and the gap between the two tells you what recomputation is costing.

Realistic values are lower than people expect. Meta reported BF16 model FLOPs utilisation between 38 and 43 percent for the Llama 3 405B run. Above 50 percent is very good, and below 25 percent usually means something is misconfigured rather than merely suboptimal.

When utilisation is poor, the cause is nearly always one of a short list, and they are separable by measurement.

Communication that failed to overlap. Sharding gathers or tensor-parallel reduces that sit exposed on the critical path instead of hiding behind compute. Visible as gaps in a profiler timeline.

The pipeline bubble. Fixed by the formula from the previous lesson, and predictable in advance rather than mysterious.

Stragglers. One slow device holds up every collective, since a collective completes only when its last participant arrives. A single throttling GPU or a degraded network interface slows the entire job to its pace, and the effect is invisible in per-device averages.

Data loading. If the input pipeline cannot keep the accelerators fed, everything downstream waits. Easy to overlook and easy to test by training on synthetic tensors.

Small kernels. Tiny per-device shapes mean launch overhead dominates and the tensor cores never reach efficiency.

The diagnostic order is: measure the per-step time breakdown first, and only then form a theory.

The most useful published account of large-run reliability comes from Meta's Llama 3 work, and its numbers reset most people's expectations.

Training the 405B model used 16,384 H100 accelerators over 54 days. Across that period there were 466 job interruptions. Of those, 47 were planned maintenance and 419 were unexpected, which works out to roughly one unplanned failure every three hours.

The breakdown is instructive. Faulty GPUs accounted for 148 interruptions, or 30.1 percent. GPU HBM3 memory accounted for 72, or 17.2 percent. Together, GPUs or their onboard memory caused about half of all failures.

And despite that, the team maintained over 90 percent effective training time.

Both halves of that matter. Component failure at this scale is continuous and unavoidable, arriving several times a day, and it is nevertheless compatible with a highly efficient run. The difference is entirely in the engineering around the failure, not in preventing it.

The design assumption to adopt is that hardware will fail during your run, repeatedly.

Checkpointing frequency is a genuine optimisation with a closed-form answer, and treating it as a guess is how runs lose days.

Two costs oppose each other. Checkpoint too often and you spend the run writing hundreds of gigabytes to storage. Checkpoint too rarely and each failure discards more work, on average half the interval.

Balancing them gives a classic first-order result: the optimal interval is the square root of twice the checkpoint cost times the mean time between failures.

With a three-hour mean time between failures and a two-minute checkpoint write, the optimal interval is about 1610 seconds, or roughly 27 minutes. Checkpoint writing then consumes about 7.5 percent of wall-clock, and each failure discards about 13 minutes of work on average.

The shape of the answer is more useful than the number. Because it is a square root, the optimum is flat: being off by a factor of two costs only a few percent. The expensive errors are the order-of-magnitude ones, checkpointing every few minutes or once a day.

Driving the checkpoint cost down moves the optimum and improves both terms at once, which is why asynchronous and sharded checkpointing are worth the engineering.

A checkpoint that does not permit exact resumption is a checkpoint that will cost you a rerun, and the omissions are predictable.

Model parameters are the obvious part and the smallest. For a 70B model in 16-bit that is 140 GB.

Optimizer states are the larger part and the one most often forgotten in ad-hoc scripts. Adam's moments are 840 GB in 32-bit for the same model. Resuming without them restarts the optimizer cold and produces a visible loss spike.

The learning rate schedule position, since resuming at the wrong point on a cosine decay silently changes the training recipe.

The data loader state, so training resumes at the right position in the corpus rather than re-consuming data it has already seen or skipping data it has not.

All random number generator states, on every rank, so dropout masks and data shuffling continue as they would have.

Under sharding, each rank writes its own shard, which parallelises the write across the cluster and is why checkpoint cost need not scale with model size in wall-clock terms.

A long run is best understood as a loop that must close, rather than as a job that runs.

Training proceeds and periodically writes a checkpoint. Something fails. The job is detected as dead, the faulty node is identified and drained, a replacement or spare is brought in, the last checkpoint is loaded on every rank, and training resumes.

Every box on that path is a place a run stalls. Detection that relies on a human noticing costs hours rather than minutes. Manual node identification costs more. A checkpoint that cannot be loaded on a different node count blocks resumption entirely, which is why checkpoint portability across cluster shapes matters.

The metric to hold the whole loop accountable is effective training time: the fraction of wall-clock spent making forward progress. Meta's over-90-percent figure was achieved while failing every three hours, which is only possible when the loop closes automatically.

The engineering target is not fewer failures. It is a shorter loop.

The characteristic pathology of a long training run is the loss spike: the curve is descending steadily, then jumps sharply, and either recovers over some steps or diverges to a useless state.

The triage question is whether it is a data problem, a numerical problem or a hardware problem, and the three have different signatures.

Data. A pathological batch, a corrupted shard, a document of repeated tokens. Identifiable because it is reproducible: replaying that exact batch reproduces the spike, and skipping it does not.

Numerical. Attention logits growing until the softmax saturates, or a gradient norm excursion. Usually visible in advance in gradient norm monitoring, and addressed with gradient clipping or normalisation placement rather than by restarting.

Hardware. A device producing wrong results without crashing. Not reproducible on a rerun, and the only clue may be that one rank's gradient norm differs from its peers.

The standard response is to roll back to the last good checkpoint and skip a window of data. Doing so blind is a mistake: log which batches were skipped, because a recurring spike at the same corpus position is a data bug that skipping merely hides.

Every failure discussed so far is loud: a process dies, a collective times out, something crashes. The dangerous category is the quiet one.

Silent data corruption is a device that computes the wrong answer and reports success. A memory cell degrades, an arithmetic unit produces an incorrect result under specific conditions, and nothing raises an error. The wrong number enters a gradient, gets averaged into the update by an all-reduce, and propagates to every rank.

This is genuinely hard because the normal signals are absent. No crash, no timeout, and the loss may only drift slightly rather than spiking. A run can degrade for hours before anyone suspects hardware.

The practical defences are indirect. Monitor per-rank gradient norms rather than only the global average, since a corrupting device often shows a norm inconsistent with its peers. Run periodic health checks that execute a known computation and compare against an expected result. And treat an inexplicable, non-reproducible loss anomaly as a hardware hypothesis worth testing rather than a mystery to be waited out.

At sixteen thousand devices, rare per-device events are frequent job-level events.

The instrumentation that matters has to exist before the run, because a long run is not an experiment you can casually repeat.

Throughput, as tokens per second and model FLOPs utilisation, per step. A drift downward over hours is the earliest signal of a straggler developing.

A per-step time breakdown separating compute, communication and data loading, so a regression can be attributed rather than guessed at.

Loss and gradient norm, logged per rank and not only globally averaged, since the per-rank view is what exposes silent corruption.

Checkpoint write duration and success, because a checkpoint system that has been quietly failing is discovered at the worst possible moment.

Hardware telemetry: temperatures, clock throttling, error-correcting code error counts, network link health. Rising correctable memory errors on a device frequently precede its failure.

And a cheap evaluation on a fixed held-out set at a regular cadence, because training loss can look healthy while the model has stopped improving on anything you care about.

The common thread is that all of these are cheap to add beforehand and impossible to add retroactively.

Distributed training rewards a small number of unglamorous habits more than any configuration insight.

Scale up in stages. Run on one device, then one node, then a few nodes, then the cluster, checking loss curves match at each step. A bug found at eight devices costs minutes; the same bug found at a thousand costs a day of cluster time.

Hold the global batch fixed when changing parallelism. Otherwise a configuration change silently becomes an optimisation change, and the comparison is meaningless.

Test resumption deliberately. Kill the job on purpose, restart from a checkpoint, and confirm the loss curve continues rather than jumping. An untested recovery path is a recovery path that does not work.

Change one degree at a time and measure, because the interactions between the three axes are not intuitive.

Write the configuration down with the run, including every parallelism degree, the global batch, and the code revision. Six weeks later nobody remembers, and the model that came out has to be explainable.

None of this is about parallelism. All of it is what makes parallelism usable.