flash-attention
3 free lessons tagged flash-attention across AI. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.
Finding the Next One: Fusion Beyond Attention
The pattern that made attention slow recurs across the stack, and once you know what to look for it is easy to find. This lesson applies the diagnosis to normalisation layers, optimizer steps, loss functions and inference decoding, covers why fused attention silently stops applying when a model deviates slightly from standard, and gives the profiling routine that decides where to look first.
Online Softmax: Tiling Across a Reduction
Softmax normalises over a whole row, which appears to require the whole row before anything can be produced, which appears to require the score matrix to exist. This lesson works through the rescaling identity that removes that obstacle, the running max and sum that make it numerically safe, and the backward pass trick of recomputing the matrix from two saved statistics.
Attention Is Memory-Bound, and Nobody Noticed for Five Years
For years attention was optimised by reducing FLOPs, and approximate methods that cut FLOPs kept failing to run faster. The reason is that attention was never compute-bound: it spends its time moving a matrix between GPU memory tiers. This lesson establishes that hierarchy, counts the traffic a standard implementation generates, and shows why an exact algorithm beat every approximation.