transformers
10 free lessons tagged transformers across AI. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.
Architectures for Order Book Data, and Why They Help Less Than Expected
Convolutional and recurrent networks have been applied to order book data with published success, and the architectures encode real assumptions about the book's structure. This lesson explains what each one assumes, why the gains over simple baselines are smaller than headline numbers suggest, and where the modelling effort is better spent.
Circuits: Reverse-Engineering Transformer Algorithms
How attention heads read and write to a shared residual stream, how they compose into circuits, and the two best-understood examples: the induction head behind in-context learning and the indirect object identification circuit in GPT-2.
Features, Directions, and Superposition
Why individual neurons are the wrong unit for understanding a neural network, and what the superposition hypothesis says is really going on inside the activations.
What Neural Language Models Can Actually Compute
The same hierarchy that classifies grammars can classify neural networks. This lesson places recurrent networks and transformers on the Chomsky ladder, explains why a transformer is theoretically weaker than it looks, and shows how formal language theory predicts where today's language models generalize and where they break.
Speculative decoding: making LLM inference faster without changing the output
How draft-then-verify decoding gets multiple tokens per forward pass of a large model, why rejection sampling makes it provably lossless, and where the draft comes from (small models, Medusa heads, self-speculation, EAGLE trees).
LLM Inference Internals: KV Cache, Sampling, and Serving at Scale
A deep dive into how large language models actually run in production — why prefill is fast and decode is slow, how the KV cache works, sampling strategies like temperature and top-p, speculative decoding, and continuous batching with vLLM.
Modern LLM Architectures: From Decoder-Only to Mixture-of-Experts
A deep tour of the architectural choices powering today's large language models — decoder-only Transformers, encoder-decoder designs, grouped-query attention, RoPE positional embeddings, and mixture-of-experts routing — with concrete numbers and trade-offs.
LLM Pretraining: Data, Loss, and What Actually Happens
A deep dive into how large language models learn from raw text: the next-token prediction objective, cross-entropy loss, the messy reality of web data curation (Common Crawl, dedup, quality filters), and the lineage from The Pile to FineWeb.
LLM Tokenization in Depth: BPE, Byte-Level BPE, and SentencePiece
A rigorous tour of how modern LLMs split text into tokens — covering byte-pair encoding, GPT-2/Llama's byte-level variant, SentencePiece, vocabulary design trade-offs, and why your tokenizer silently determines multilingual fairness, code quality, and arithmetic ability.
Attention and Transformers
From the limits of RNNs to the self-attention mechanism that replaced them. Learn how queries, keys, and values implement scaled dot-product attention, why multi-head attention captures richer structure, how positional encodings inject order, and how all of this assembles into a transformer block.