long-context

Pure recurrent models match transformers on broad language benchmarks and lose on the tasks that need exact retrieval from context. That is a capacity limit, not an engineering gap: a fixed state holds a fixed amount. This lesson covers what the evaluations showed, why a handful of attention layers recovers almost all of it, how Jamba and Samba are laid out, and when this is worth adopting.

The other route starts in control theory. A discretised linear system is a recurrence, and time-invariance turns it into one convolution, which trains in parallel. This lesson follows that line: why HiPPO initialisation matters, why time-invariance is what stops the model choosing what to remember, how selection breaks the convolution, and how the parallel scan gets it back.

Attention costs two separate things people conflate: quadratic compute during training, and a cache that grows without bound during inference. At a million tokens that cache is 344 GB while a recurrent state is 16.8 MB and constant. This lesson separates the two costs, shows why the classical RNN alternative failed, and states the trilemma every architecture since has been negotiating.

A capability-by-capability tour of frontier LLMs in 2026 — which models are strong at reasoning, code, math, long-context, multilingual, multimodal, and tool use, with hedged comparisons instead of point-estimate benchmark wars.

What the context window actually is, why 1M-token marketing doesn't mean what you think, the cost and accuracy curves that bend as you stuff more in, and the strategies (compaction, caching, retrieval) that keep long-context apps sane.