The Recall Tradeoff, and Why Everything Became a Hybrid

By the time selective state space models reached the scale where comparison was meaningful, the result was consistent and specific.

Mamba and Mamba-2 match transformers on many standard benchmarks at around 8 billion parameters. Perplexity is competitive, general language understanding holds up, and the efficiency gains are real.

They lag on a narrow and identifiable set of tasks. Five-shot MMLU is one. Phonebook lookup is the cleanest: give the model a list of names and numbers in context, ask for one number, and see whether it comes back. Transformers do this reliably. State space models struggle.

More broadly, the gap appears wherever the task is copying from or retrieving out of the context rather than reasoning about it in aggregate.

The pattern in the failures is more useful than the aggregate scores. This is not a model that is generally weaker. It is a model that is equally good at almost everything and specifically bad at exact retrieval, which is precisely what the previous two lessons predicted from the capacity argument.

The five-shot result deserves separating out, because it looks like a reasoning failure and is not.

In a few-shot prompt, examples are placed in context and the model must apply their pattern to a new input. That requires holding the examples available and referring back to them precisely, including their exact formatting and label vocabulary.

Attention does this by construction. Every example remains addressable in the cache, so referring back is a lookup.

A fixed state has to have compressed those examples into its capacity alongside everything else. The general shape survives compression; exact details do not, and few-shot prompting depends on exact details.

The operational consequence is worth stating clearly. A model that appears fine in zero-shot use can degrade specifically when you add examples to improve it, which is a counter-intuitive failure that is easy to misattribute.

The same reasoning covers a large family of real workloads: retrieval-augmented generation with long retrieved passages, agent trajectories referring to earlier tool output, and any prompt where the answer is quoted from the input rather than derived.

The most useful framing of this comes from work on the recall-throughput tradeoff, which established that the relationship between state size and recall ability is continuous.

Efficient alternatives to attention maintain a fixed-size recurrent state and struggle at recall in proportion to how small that state is. Enlarge the state and recall improves; enlarge it enough and you approach attention's recall along with attention's cost.

Crucially, this is a frontier that can be traversed. Varying the linear attention feature dimension and the size of any attention window moves you along a curve, recovering the full quality of attention at one end and the small state of the alternatives at the other.

That reframes the design question. It is not whether to use attention or a recurrent state. It is where on the curve your workload should sit, and the answer depends on how much exact retrieval the task actually requires.

And it explains why the honest comparison is never architecture against architecture. It is quality at a given inference memory budget, which is the axis the frontier is drawn on.

The empirical discovery that settled the field's direction is that the mixture is enormously better than either pure form, and it does not take much attention.

A model that is mostly recurrent layers with occasional attention layers interleaved recovers most of the retrieval ability while keeping most of the efficiency. The attention layers provide exact addressable access where it is needed; the recurrent layers do the bulk of the work at constant cost.

The theoretical side supports this. A two-layer hybrid combining one recurrent layer with one attention layer can solve selective copying tasks with an embedding dimension only logarithmic in the vocabulary size and sequence length, which is a strong statement about how little attention is required to restore the capability.

The intuition is that the two mechanisms are complementary rather than redundant. Compression handles the aggregate: topic, syntax, the general state of the discourse. Direct addressing handles the specific: this number, this name, this exact token.

Most of what a language model does is aggregate. The specific matters intensely when it matters at all, which is exactly the profile that suits a small number of dedicated layers.

Given that hybrids work, the design question reduces largely to how much attention and where.

Too few attention layers and retrieval degrades toward the pure recurrent behaviour. Too many and the KV cache grows back, taking the efficiency gain with it, since a single attention layer over a million tokens still caches a million tokens.

A ratio around seven recurrent layers to one attention layer appears near-optimal on the compute-quality frontier, and that figure recurs across published designs.

What the ratio controls is precisely the inference memory budget. With one attention layer in eight, the cache is one eighth the size of a full transformer's at the same depth and context, which is what makes very long contexts tractable.

Placement matters alongside count, and is less settled. Attention layers spread evenly through the depth is the common choice. Some designs put none in the first few layers, on the reasoning that early layers handle local structure the short convolution already covers.

This is also a knob you can tune per deployment rather than accept from a paper, since the right ratio depends on how retrieval-heavy your workload is.

Published hybrids differ in how they combine the two mechanisms, and the three shapes have different characters.

Interleaving alternates whole layers. Most layers are recurrent, a minority are attention, and the ratio is the main dial. Jamba takes this approach, combining Mamba and attention at roughly seven to one and adding mixture-of-experts layers to scale capacity, reaching very long contexts with substantially lower memory than a comparable transformer.

Combining within a layer gives every layer both mechanisms. Samba pairs selective state space layers with sliding window attention, so each layer has both a compressed long-range summary and exact access to a recent window. Hymba runs attention heads and state space heads side by side inside one layer, with the attention heads providing high-resolution recall and the state space heads efficient summarisation.

The distinction that matters operationally is where the cache lives. Interleaved designs concentrate it in a few layers. Within-layer designs with a window bound it everywhere. Both give a cache that is a fraction of a full transformer's, by different routes.

The combination of a recurrent state with sliding window attention deserves its own attention, because the two failure modes are complementary in an unusually clean way.

Sliding window attention gives exact access to the last w tokens and nothing before. Its weakness is entirely long-range.

A recurrent state gives compressed access to everything, with detail degrading as history accumulates. Its weakness is entirely short-range precision under load.

Put together, the window handles recent detail exactly while the state carries long-range summary, and each covers what the other lacks. The cache is bounded by the window size, so memory is constant in sequence length, and long-range information still flows through the state rather than being discarded.

This is a better arrangement than it might sound, because recency and precision are correlated in practice. The tokens whose exact form matters most are usually recent: the current instruction, the last tool result, the code being edited.

When exact recall of something distant is genuinely required, this arrangement still struggles, which is why designs that combine a window with a few full-attention layers exist.

Standard benchmarks will not tell you whether a hybrid suits your workload, because the gap lives in a place aggregate scores average away.

Test retrieval explicitly and at your context lengths. A phonebook or needle-in-a-haystack task, where a specific fact is planted at a known depth and then requested, isolates exactly the capability at issue. Vary the depth and vary the total length, since failures appear as a function of both.

Test few-shot behaviour separately from zero-shot, since the previous steps established that few-shot prompting is a retrieval task in disguise.

Test at the concurrency you will serve. The efficiency argument is largely about cache size, and cache size becomes decisive when many sequences are resident at once, so a single-stream benchmark understates the benefit substantially.

And compare at equal inference memory rather than at equal parameter count. A hybrid that matches a transformer's quality using a quarter of the cache is the result that matters, and equal-parameter comparisons obscure it in both directions.

That last point is the one most published comparisons get wrong, and it is worth insisting on when reading them.

The honest answer for most teams is not yet, and knowing why is more useful than enthusiasm.

The transformer ecosystem is enormous. Serving stacks, quantization toolchains, fine-tuning libraries, and the accumulated operational knowledge of the last several years all assume attention. A hybrid architecture means less mature tooling and fewer people who have debugged it.

The efficiency gain only matters if cache memory is your binding constraint. At 4k context and modest concurrency it is not, and adopting a less-supported architecture buys nothing.

Where the case is strong is specific. Very long contexts, above roughly 100k tokens, where the cache dominates the memory budget. High-concurrency serving where cache capacity sets how many users fit on a device. On-device inference with a hard memory ceiling. And long-running streaming workloads where an unbounded cache is disqualifying regardless of anything else.

The reason to understand the mechanism now, even without adopting it, is that the ideas are arriving inside otherwise conventional models. Sliding windows, bounded caches and gated linear recurrences are appearing as components rather than as whole architectures, and recognising them is the practical payoff.

Five conclusions, in the order the path built them.

Attention's cost is a cache that grows with the sequence, and that cache is what buys perfect recall. The two cannot be separated, because retrieval without storage is not available.

Removing the softmax permits rebracketing the product, which turns attention into a recurrent network with a matrix-valued state and two equivalent computation modes, parallel for training and recurrent for generation.

The state space line reaches the same object from control theory, with time-invariance supplying parallel training and selection supplying the content-dependence language requires, and the duality result shows the two lines describe one mechanism.

A fixed state has fixed capacity, so exact retrieval degrades with sequence length. That is a limit rather than a defect, and it shows up sharply on copying, retrieval and few-shot tasks while leaving broad language modelling largely intact.

And a small number of attention layers restores most of what compression loses, which is why the field converged on hybrids rather than on a replacement. The remaining question is a ratio, and a ratio is a thing you can tune.