Beyond Attention: Linear Attention, State Space Models, and Hybrids

At a million tokens a transformer's KV cache is 344 GB. A recurrent state doing the same job is 16.8 MB, and constant at any length. This path explains how that is possible and what it costs. Delete the softmax and attention becomes a recurrent network you can still train in parallel. Start instead from control theory and you arrive at the same object, which is the result Mamba-2 made precise. Then the honest part: a fixed state has fixed capacity, exact retrieval is what breaks first, and a few attention layers buy most of it back.