distributed-training

A configuration that fits is not a run that finishes. Llama 3 405B training saw 419 unexpected interruptions in 54 days on 16,384 GPUs, one every three hours, and still kept over 90 percent effective training time. This lesson covers model FLOPs utilisation, the optimal checkpoint interval and where it comes from, loss-spike triage, silent data corruption, and the habits that finish runs.

Sharding distributes copies; tensor and pipeline parallelism split the computation. This lesson covers the column-then-row trick that lets a transformer block communicate only twice per layer, sequence parallelism for the parts tensor parallelism cannot reach, the pipeline bubble and why it is (p-1)/m, what 1F1B and interleaving actually fix, and how the three axes compose into a 3D layout.

Data parallelism keeps N identical copies of everything. ZeRO removes that redundancy in three stages, and stage 3 takes a 70B model from 1120 GB per GPU to 17.5 GB across 64. This lesson covers what each stage shards, the communication each costs, how FSDP implements it with prefetch and wrapping policy, when offload is worth it, and why stage 3 is not automatically right.

Training memory is dominated by things that are not the model. Mixed-precision Adam costs 16 bytes per parameter, so a 70B model needs 1120 GB of state before one activation is stored. This lesson works through where every byte goes, why data parallelism helps throughput but not memory, how accumulation and recomputation trade compute for space, and what each axis of parallelism addresses.

Why local training on non-identical data pulls clients apart, how averaging their updates produces a model that suits nobody, and the control-variate fix that corrects the drift.

The federated setting: why hospitals, phones, and banks cannot pool their data, what changes when the training loop crosses a network, and the FedAvg algorithm that made the idea practical.