inference

Recursive formats need a machine with a stack, and a stack breaks the precomputed index because the mask now depends on context. This lesson covers context-free grammars and pushdown automata, XGrammar's split between context-independent and context-dependent tokens, why overlapping grammar work with the GPU makes overhead near zero, and how this interacts with batching and speculative decoding.

The naive mask costs 64 million validity checks per response. The trick that made constrained decoding practical is to notice that the answer depends only on the automaton's current state, so it can be computed once per state and looked up thereafter. This lesson builds that idea: regex to DFA, why the state is a sufficient summary, how JSON Schema compiles down, and what the index costs.

Prompting for a format gives a high success rate, and a high success rate is not a guarantee. This lesson locates the one place in the decoding loop where a guarantee is possible, shows what masking logits does to the probability distribution, works through why a 5 percent failure rate destroys tail latency rather than average latency, and separates the three families of structured output.

Quantization damage does not show up where people look for it. Perplexity barely moves while hard tasks degrade, and long reasoning suffers most because error compounds. This lesson covers building an evaluation that detects real loss, where the published cliffs are, KV cache quantization as a separate lever, end-to-end memory sizing, and the rollout that catches what evals miss.

Round-to-nearest minimises the wrong thing. The algorithms that make 4-bit weights usable minimise output error instead, using a small calibration set. This lesson works through GPTQ's Hessian-guided error compensation and AWQ's activation-aware channel scaling, why AWQ tolerates calibration mismatch better, how much data you need, and where QAT and QLoRA fit.

Round-to-nearest works on small models and falls apart on large ones, because beyond a certain scale transformers grow systematic activation outliers in a few fixed channels, a hundred times larger than everything else. This lesson shows arithmetically why one outlier destroys a tensor, then works through the three families of fix: decompose, smooth, and rotate.

Decoding is memory-bandwidth-bound, so fewer bits per weight means more tokens per second, not fewer. This lesson builds the mechanism from the arithmetic up: the affine mapping, a worked example done by hand, why group size costs fractional bits, the difference between W4A16 and W8A8, and which formats the hardware actually accelerates.

Belief propagation, the cavity method, and approximate message passing: how physics-derived algorithms achieve the best performance any efficient method can, and how state evolution predicts their behaviour exactly before you run them.

The central result of the field: problems split into three regimes as data increases, and the middle one contains enough information to solve them while no efficient algorithm can. Community detection and planted clique make it concrete.

Why statistical physics has anything to say about algorithms: planted models with known ground truth, the large-system limit where randomness stops fluctuating, and the Bayes-optimal benchmark that makes hardness measurable.

A model that runs is not a model that serves. This lesson covers what an inference server does that a naive loop cannot, continuous batching and why it dominates throughput, the memory arithmetic that decides which hardware you need, quantization for serving, and how to size a deployment from a traffic estimate.

Energy-based models replace input-to-output functions with a scalar energy that scores how compatible a configuration is. This lesson covers the energy function, inference as finding the lowest-energy answer, the energy landscape picture, the link from energy to probability via the partition function, and why this framing handles problems with many valid answers.

Modern AI got better mainly by getting bigger, and bigger means more computation, which is why the story of AI is now a story about hardware, power, and money. Learn what compute actually is, why scaling laws made more of it pay off so reliably, the difference between training and inference demand, and why this buildout is unlike previous technology booms.

How draft-then-verify decoding gets multiple tokens per forward pass of a large model, why rejection sampling makes it provably lossless, and where the draft comes from (small models, Medusa heads, self-speculation, EAGLE trees).

A deep dive into how large language models actually run in production — why prefill is fast and decode is slow, how the KV cache works, sampling strategies like temperature and top-p, speculative decoding, and continuous batching with vLLM.

How two landmark papers — Kaplan et al. 2020 and DeepMind's Chinchilla 2022 — rewrote our understanding of compute-optimal training, why the industry now deliberately overtrains models, and how inference costs flip the math entirely.

Understand what it really means to update beliefs with data. Derive Bayes' theorem from first principles, dissect the roles of prior, likelihood, posterior, and evidence, work through a complete Beta-Binomial conjugate example numerically, and see why the base-rate fallacy trips up even experts.