quantization

Quantization damage does not show up where people look for it. Perplexity barely moves while hard tasks degrade, and long reasoning suffers most because error compounds. This lesson covers building an evaluation that detects real loss, where the published cliffs are, KV cache quantization as a separate lever, end-to-end memory sizing, and the rollout that catches what evals miss.

Round-to-nearest minimises the wrong thing. The algorithms that make 4-bit weights usable minimise output error instead, using a small calibration set. This lesson works through GPTQ's Hessian-guided error compensation and AWQ's activation-aware channel scaling, why AWQ tolerates calibration mismatch better, how much data you need, and where QAT and QLoRA fit.

Round-to-nearest works on small models and falls apart on large ones, because beyond a certain scale transformers grow systematic activation outliers in a few fixed channels, a hundred times larger than everything else. This lesson shows arithmetically why one outlier destroys a tensor, then works through the three families of fix: decompose, smooth, and rotate.

Decoding is memory-bandwidth-bound, so fewer bits per weight means more tokens per second, not fewer. This lesson builds the mechanism from the arithmetic up: the affine mapping, a worked example done by hand, why group size costs fractional bits, the difference between W4A16 and W8A8, and which formats the hardware actually accelerates.

Three ideas carry almost all production vector search. This lesson covers HNSW as a navigable small-world graph with its M and ef parameters, IVF as coarse partitioning with nprobe, and the quantization schemes that cut memory by an order of magnitude, plus how they compose into the hybrid indexes real systems actually run.