infrastructure

A configuration that fits is not a run that finishes. Llama 3 405B training saw 419 unexpected interruptions in 54 days on 16,384 GPUs, one every three hours, and still kept over 90 percent effective training time. This lesson covers model FLOPs utilisation, the optimal checkpoint interval and where it comes from, loss-spike triage, silent data corruption, and the habits that finish runs.

Data parallelism keeps N identical copies of everything. ZeRO removes that redundancy in three stages, and stage 3 takes a 70B model from 1120 GB per GPU to 17.5 GB across 64. This lesson covers what each stage shards, the communication each costs, how FSDP implements it with prefetch and wrapping policy, when offload is worth it, and why stage 3 is not automatically right.

Training memory is dominated by things that are not the model. Mixed-precision Adam costs 16 bytes per parameter, so a 70B model needs 1120 GB of state before one activation is stored. This lesson works through where every byte goes, why data parallelism helps throughput but not memory, how accumulation and recomputation trade compute for space, and what each axis of parallelism addresses.

Self-hosting is a permanent operational responsibility rather than a project. This lesson covers the security surface a local model creates, model upgrades and why they are harder than they look, capacity and cost control, what to monitor, and the honest signals that the deployment should be retired.

A model that runs is not a model that serves. This lesson covers what an inference server does that a naive loop cannot, continuous batching and why it dominates throughput, the memory arithmetic that decides which hardware you need, quantization for serving, and how to size a deployment from a traffic estimate.

Self-hosting is usually justified on privacy grounds, and the argument is often weaker than assumed. This lesson examines what actually stays private, which regulatory requirements genuinely demand it, the reasons that hold up better than privacy, and the honest total cost against an API.

The surprise of the AI buildout is that the scarcest resource is not chips or money but electricity, and the grid was not built for this. Learn why power became the binding constraint, why connecting a big new load takes years, why data centers are turning to gas and nuclear, and why an unglamorous device, the transformer, quietly gates the whole thing.

An AI data center is not a warehouse of ordinary computers; it is a single supercomputer built from tens of thousands of specialized chips wired together. Learn why GPUs beat regular processors for AI, why the network between chips matters as much as the chips, why one company dominates the market, and the cost structure that makes these buildings so expensive.

Modern AI got better mainly by getting bigger, and bigger means more computation, which is why the story of AI is now a story about hardware, power, and money. Learn what compute actually is, why scaling laws made more of it pay off so reliably, the difference between training and inference demand, and why this buildout is unlike previous technology booms.