efficiency
3 free lessons tagged efficiency across AI. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.
State Space Models: From S4 to Selection
The other route starts in control theory. A discretised linear system is a recurrence, and time-invariance turns it into one convolution, which trains in parallel. This lesson follows that line: why HiPPO initialisation matters, why time-invariance is what stops the model choosing what to remember, how selection breaks the convolution, and how the parallel scan gets it back.
Linear Attention: Removing the Softmax Buys Associativity
Softmax is the one operation forcing attention to materialise an n-by-n matrix. Remove it and associativity lets you rebracket the product so a fixed-size quantity is maintained instead. This lesson derives that step, shows why the result is a recurrent network with a matrix-valued state, explains the two computation modes, and is honest about what the softmax was doing.
What Attention Costs, and the Trilemma Underneath
Attention costs two separate things people conflate: quadratic compute during training, and a cache that grows without bound during inference. At a million tokens that cache is 344 GB while a recurrent state is 16.8 MB and constant. This lesson separates the two costs, shows why the classical RNN alternative failed, and states the trilemma every architecture since has been negotiating.