The two-stage funnel: retrieval then ranking

Open any feed and a system just did something absurd. It had a corpus of maybe a hundred million items. It had to choose roughly ten. It had about 100 milliseconds. And it had to do it for every user at once.

Score every item with a good model and the arithmetic kills you immediately. If one scoring pass takes 1 millisecond, then 100 million items is 100,000 seconds, which is over a day, for one refresh.

So no production recommender ever scores everything. That is the single fact that explains the architecture of every feed you use. The system must throw away almost the entire corpus before it thinks carefully, and the interesting engineering is entirely in how you throw things away cheaply without losing the good ones.

The standard answer is a funnel: a couple of stages, each smaller and smarter than the last.

The architecture that became standard, popularised by a 2016 paper describing YouTube's recommender, splits the job into two neural networks with very different jobs.

Candidate generation, also called retrieval, takes millions of items and returns a few hundred. It has to be extremely cheap per item, so it is allowed to be crude. Its only real duty is recall: do not lose the good stuff.

Ranking takes those few hundred and puts them in order. It runs a much heavier model with far more features, because it only has to score hundreds of things, not millions. Its duty is precision: get the top few exactly right.

The stages are graded: cheap and approximate first, expensive and accurate last.

The trick that makes retrieval cheap is to stop comparing and start measuring distance.

Represent every item as a point in a few hundred dimensions, an embedding. Represent the user as a point in the same space. Now define the whole problem as: find the points nearest to the user's point.

That reframing is the payoff. Similarity is no longer a model call per item, it is a vector operation. The learning problem becomes: place things so that the geometry is meaningful, so that items a user would enjoy end up physically close to that user's vector.

YouTube's paper framed candidate generation as extreme multiclass classification: given this user, predict which video out of millions they watch next. Training that directly needs a softmax over the entire corpus, which is far too expensive, so it is trained with negative sampling instead: compare the true item against a handful of random ones rather than against all of them.

The structure that produces those embeddings is the two-tower model. Two separate networks that never touch until the very end.

# user tower: everything about the person
user_vec = user_tower([
    watch_history_embeddings,   # what they watched
    search_tokens,              # what they searched
    geo, device, age_of_account,
])                              # -> [0.21, -0.03, ...] 256 numbers

# item tower: everything about the thing
item_vec = item_tower([
    content_embedding,          # what it IS (audio, visual, text)
    creator_id, language, topic, age_of_item,
])                              # -> [0.19, -0.01, ...] same 256 dims

score = dot(user_vec, item_vec)   # closeness = predicted affinity

The towers are trained together so that a dot product between them predicts engagement. But they run apart, and that separation is the entire point of the design.

Here is the constraint that forces the architecture. A model that takes user and item together is more accurate, because it can learn interactions between them. But it must be run once per pair. Millions of items means millions of forward passes per request. Impossible.

Because the two towers never see each other, an item's vector does not depend on who is asking. So:

1. Compute every item's vector offline, in advance, once.
2. Load them into an index.
3. At request time run the user tower once, then look up neighbours.

That is millions of forward passes traded for one. The cost of a request stops scaling with the size of the corpus.

The price is real: the towers cannot model user-item interactions, only their positions. That crudeness is acceptable precisely because retrieval is not the final word. Ranking, which does see both together, cleans it up afterwards. The funnel lets each stage be wrong in ways the next stage can fix.

Even with precomputed vectors, comparing the user against a hundred million item vectors on every request is still too slow. Exact nearest neighbour search is linear in corpus size, and linear is exactly what we are trying to escape.

So the system does not find the nearest neighbours. It finds approximately the nearest neighbours, in roughly logarithmic time, using an ANN index.

The idea is to precompute structure over the vectors: partition the space into regions, or build a navigable graph across the points, so a query can walk to a good neighbourhood while touching only a tiny fraction of the corpus.

The tradeoff is explicit and tunable: search fewer regions and you get answers in microseconds while occasionally missing a genuinely close item. Search more and you get better recall for more compute.

That missed item is usually invisible. When you are picking 500 candidates out of 100 million to hand to a ranker, whether the true 400th-best made the cut changes nothing a user would notice.

Now the budget is different. A few hundred candidates instead of a hundred million means roughly six orders of magnitude more compute available per item. The ranker spends it.

It takes user and item together, plus features retrieval could never afford: how many times this exact user saw this creator this week, time since they last watched something similar, current session length, item freshness, device, and dozens more. It can model interactions, because it sees both sides at once.

One detail from the YouTube paper matters more than the architecture. They did not train the ranker to predict clicks. They modified logistic regression to predict expected watch time.

The reason is a lesson about optimisation targets generally: optimise for click probability and the system learns that clickbait works, because a misleading thumbnail genuinely does maximise clicks. The model was not broken. It was succeeding at the objective it was given, and the objective was wrong.

The two-stage funnel has been standard for a decade and still runs most of what you use. It is now being challenged, and the challenge is already in production.

The reframing: stop treating recommendation as retrieve then score and treat it as sequence prediction. Given everything a user has done, generate what they do next, the way a language model generates a next token. Items are assigned semantic IDs, discrete codes derived from their content, so a transformer can emit the ID of the next item directly instead of scoring a candidate list.

Meta published the strongest evidence. Their HSTU architecture (Zhai et al., ICML 2024) recasts recommendation as sequential transduction, scales to 1.5 trillion parameters, and reported a 12.4 percent improvement in online A/B tests, deployed across multiple surfaces of a platform with billions of users. The code is open source.

Others go further: Kuaishou's OneRec (2025) collapses retrieval and ranking into a single generative model, attacking the two-stage split itself.

So learn the funnel, because it is what almost everything runs. Just do not mistake it for the end of the story.

Item vectors are built offline and indexed. At request time the user tower runs once, nearest-neighbour search cuts millions to hundreds, and only then does the expensive ranker look at user and item together.