Lessons

Speech recognition's hard problem is that audio and text have different lengths and nobody labelled which frame goes with which letter. CTC, RNN-T and attention encoder-decoders are three answers, and which one a system uses decides whether it can stream. This lesson covers all three, why Whisper's weak supervision worked, and the failure that follows from a recogniser containing a language model.

Before a model can process speech it has to be discretised, and audio resists that harder than text does. This lesson covers why raw waveforms are the wrong representation, how neural audio codecs learn a discrete one, what residual vector quantization actually does, and the arithmetic that governs every speech model: a minute of talking is around 195 text tokens or 36,000 audio tokens.

The exchange is a case study with transferable rules. A conclusion resting on failures needs a failure taxonomy. Every instance must be verified solvable before anyone is scored against it. Output format is a confound whenever answers get long. And when two explanations fit the same data, the productive move is to find the prediction on which they differ, then test it.

The response disputed none of the data and argued the experiment measured something other than reasoning. Models had to print move lists exceeding their output limits, and said so in the transcripts. Some instances had no solution and were scored as failures anyway. And asking for a program instead of a move list produced high accuracy on instances reported as total collapse.

Apple researchers built an evaluation designed to fix a real problem with benchmarks: puzzles where difficulty turns up smoothly while the logic stays identical, and every step can be checked. They found accuracy collapsing to zero past a threshold, and not improving when the solution algorithm was handed to the model. This lesson covers the design and why it was a good one.

A model went from 27 percent to around 57 percent, so it is more than halfway to AGI and the rest arrives shortly. That inference is wrong in at least four ways, and working through why is more useful than the score itself. This lesson covers the linearity assumption, construct validity, contamination, and what the framework is good for once you stop reading it as a progress bar.

The framework scores ten cognitive domains at ten percent each. Running it produces something more useful than the headline totals of 27 percent for GPT-4 and around 57 for GPT-5: a jagged profile, where a model is at or near full marks on some domains and at zero on others. This lesson walks the domains, reads both profiles column by column, and shows what the jaggedness explains.

For a term that anchors company charters, safety policy and hundreds of billions in investment, AGI has been remarkably undefined. A 2025 paper with over thirty authors proposes fixing that by borrowing psychology's most validated framework for human intelligence. This lesson covers why the definition was missing, what the paper anchors to, and what the borrowing assumes.

The paper's answer to safety is that reward signals adapt to human feedback: a network inside the reward function learns which grounded signals to weight from how people respond. This lesson works through that proposal and the published critique arguing it fails twice over, on specification gaming and on goal misgeneralization, and what an honest reading of the disagreement leaves you with.

The paper is concrete about what an experiential agent would differ on, and names four: it lives in a continuous stream rather than episodes, acts in the world rather than emitting text, takes rewards from grounded signals rather than human judgement, and plans in terms it worked out rather than imitating human chain of thought. This lesson works through each.

David Silver and Richard Sutton argue that the current approach has a ceiling built into it, because a system trained to predict what humans wrote is aiming at human performance by construction. This lesson works through their three eras, the claim about data exhaustion, why they think superhuman performance needs a different learning signal, and the honest counter-arguments.

The pattern that made attention slow recurs across the stack, and once you know what to look for it is easy to find. This lesson applies the diagnosis to normalisation layers, optimizer steps, loss functions and inference decoding, covers why fused attention silently stops applying when a model deviates slightly from standard, and gives the profiling routine that decides where to look first.

The reason most teams never fuse anything is that CUDA asks you to manage threads, shared memory and synchronisation by hand. Triton moves the unit of programming from a thread to a block and hands the rest to a compiler. This lesson covers what that buys, what it still asks of you, how to decide a kernel is worth writing, and how to be sure it is correct.

Softmax normalises over a whole row, which appears to require the whole row before anything can be produced, which appears to require the score matrix to exist. This lesson works through the rescaling identity that removes that obstacle, the running max and sum that make it numerically safe, and the backward pass trick of recomputing the matrix from two saved statistics.

For years attention was optimised by reducing FLOPs, and approximate methods that cut FLOPs kept failing to run faster. The reason is that attention was never compute-bound: it spends its time moving a matrix between GPU memory tiers. This lesson establishes that hierarchy, counts the traffic a standard implementation generates, and shows why an exact algorithm beat every approximation.

Pure recurrent models match transformers on broad language benchmarks and lose on the tasks that need exact retrieval from context. That is a capacity limit, not an engineering gap: a fixed state holds a fixed amount. This lesson covers what the evaluations showed, why a handful of attention layers recovers almost all of it, how Jamba and Samba are laid out, and when this is worth adopting.

The other route starts in control theory. A discretised linear system is a recurrence, and time-invariance turns it into one convolution, which trains in parallel. This lesson follows that line: why HiPPO initialisation matters, why time-invariance is what stops the model choosing what to remember, how selection breaks the convolution, and how the parallel scan gets it back.

Softmax is the one operation forcing attention to materialise an n-by-n matrix. Remove it and associativity lets you rebracket the product so a fixed-size quantity is maintained instead. This lesson derives that step, shows why the result is a recurrent network with a matrix-valued state, explains the two computation modes, and is honest about what the softmax was doing.

Attention costs two separate things people conflate: quadratic compute during training, and a cache that grows without bound during inference. At a million tokens that cache is 344 GB while a recurrent state is 16.8 MB and constant. This lesson separates the two costs, shows why the classical RNN alternative failed, and states the trilemma every architecture since has been negotiating.

Training on your own model's output degrades it, and the widely cited result rests on an assumption people rarely notice: that synthetic data replaces real data. Follow-up work showed accumulating synthetic alongside real bounds the error instead. This lesson covers both stages of collapse, why the rule is accumulate rather than replace, how to set a mixing ratio, and the evaluation discipline.

Generation is the cheap half. What you discard determines what the student learns. This lesson orders the filters by strength: machine verification where an answer can be checked, self-consistency where it cannot, LLM-as-judge with its known position and length biases, and cheap heuristics. It ends on contamination, the failure that invalidates results rather than degrading them.

Prompt a model for a thousand examples and you get one example a thousand times, with the nouns changed. This lesson covers why raising temperature does not fix that, the seed-conditioning trick that does, how Self-Instruct and Evol-Instruct systematise it, programmatic generation where ground truth is known by construction, and how to measure whether a dataset actually covers its input space.

Synthetic data is roughly three orders of magnitude cheaper than human annotation, and cheapness is the least interesting thing about it. This lesson establishes what generation can and cannot manufacture: it produces coverage and format, never information the generator lacks, and the one exception is verifiable domains, where a checker turns generation into search.

Constrained decoding guarantees syntax and can degrade reasoning. Published work found strict formats hurt reasoning tasks while helping classification, because a schema demanding the answer first denies the model room to derive it. This lesson covers where the damage comes from, why field order is thinking order, the enum trap that manufactures confident errors, and how to measure the cost.