deployment

Quantization damage does not show up where people look for it. Perplexity barely moves while hard tasks degrade, and long reasoning suffers most because error compounds. This lesson covers building an evaluation that detects real loss, where the published cliffs are, KV cache quantization as a separate lever, end-to-end memory sizing, and the rollout that catches what evals miss.

Decoding is memory-bandwidth-bound, so fewer bits per weight means more tokens per second, not fewer. This lesson builds the mechanism from the arithmetic up: the affine mapping, a worked example done by hand, why group size costs fractional bits, the difference between W4A16 and W8A8, and which formats the hardware actually accelerates.

What happens when a prediction model leaves the benchmark: adversarial fragility, distribution shift, the feedback loop where your own robot changes the behaviour it predicts, and how planners consume uncertainty safely.

Self-hosting is usually justified on privacy grounds, and the argument is often weaker than assumed. This lesson examines what actually stays private, which regulatory requirements genuinely demand it, the reasons that hold up better than privacy, and the honest total cost against an API.