FlashAttention and the Art of Finding the Real Bottleneck

For five years attention was optimised by cutting FLOPs, and the faster algorithms kept not being faster. They were solving the wrong problem: attention spends its time moving a matrix between GPU memory tiers, not computing. This path follows the diagnosis and the fix, from the memory hierarchy through the rescaling identity that lets softmax be tiled, to writing fused kernels yourself, and ends where the same signature shows up next: normalisation, the optimizer step, and the logits tensor nobody thinks about.