state-space-models

Pure recurrent models match transformers on broad language benchmarks and lose on the tasks that need exact retrieval from context. That is a capacity limit, not an engineering gap: a fixed state holds a fixed amount. This lesson covers what the evaluations showed, why a handful of attention layers recovers almost all of it, how Jamba and Samba are laid out, and when this is worth adopting.

The other route starts in control theory. A discretised linear system is a recurrence, and time-invariance turns it into one convolution, which trains in parallel. This lesson follows that line: why HiPPO initialisation matters, why time-invariance is what stops the model choosing what to remember, how selection breaks the convolution, and how the parallel scan gets it back.

Softmax is the one operation forcing attention to materialise an n-by-n matrix. Remove it and associativity lets you rebracket the product so a fixed-size quantity is maintained instead. This lesson derives that step, shows why the result is a recurrent network with a matrix-valued state, explains the two computation modes, and is honest about what the softmax was doing.

Attention costs two separate things people conflate: quadratic compute during training, and a cache that grows without bound during inference. At a million tokens that cache is 344 GB while a recurrent state is 16.8 MB and constant. This lesson separates the two costs, shows why the classical RNN alternative failed, and states the trilemma every architecture since has been negotiating.