The Two-Stage Machine: Why No Model Ranks the Whole Catalogue

A recommendation surface, a homepage row, a feed, an autoplay next-up, has one job: from a catalogue of items, select and order the handful this user should see right now.

Stated like that, it sounds like ranking. The constraints are what make it an architecture problem:

• The catalogue is huge. Millions of videos, tens of millions of tracks, hundreds of millions of listings.
• The budget is tiny. The recommendations must arrive within a page load, so the whole computation gets tens of milliseconds, not seconds.
• Quality needs expensive models. Telling apart two plausible items for this user takes a model that looks at many features of the user, the item, and the context together, and such models cost real compute per item scored.

Multiply it out: an expensive model, times millions of items, times a millisecond budget, does not go. Something has to give, and what gives is the assumption that one model does the whole job.

Key idea: recommendation at scale is the art of spending almost nothing per item on millions of items, so you can afford to spend heavily on a few hundred. Every large system, whatever its domain, converges on this funnel.

The architecture that resolves the arithmetic was described openly in Covington, Adams and Sargin's 2016 paper on YouTube's recommender, and it remains the template the industry teaches from.

The two stages answer different questions on purpose. Candidate generation answers "what is plausibly relevant to this user?", a recall problem: its sin is missing something good, not including something mediocre. Ranking answers "of these plausible items, which are best, in what order?", a precision problem: its sin is ordering badly.

The division of labour also divides the engineering. Retrieval must be fast over everything, so it leans on structures built offline: precomputed embeddings, indexes, cached lists. Ranking must be smart over a little, so it runs live models with fresh features. Two teams, two serving systems, two failure modes, one funnel.

Put numbers on the funnel to see why it is not optional. Suppose the ranking model costs 100 microseconds of compute per item scored, modest for a feature-rich model, and the surface has a 50 millisecond budget for the whole recommendation call.

This is the same funnel economics as hybrid search re-ranking from the search world, and the same moral: the cheap stage is not the dumb stage, it is the stage that decides what is possible. Teams routinely over-invest in the ranker, where progress is measurable and glamorous, and under-invest in retrieval coverage, where the silent losses live.

Production candidate generation is not one algorithm but a committee. Each source is cheap, biased in a known direction, and blind in a known direction, and the union covers what no single source can.

Typical members: an embedding-similarity source proposing items near the user's taste vector, the subject of the next lesson; a co-occurrence source proposing items frequently consumed alongside what the user just consumed; a freshness source injecting new items that have no history yet; a popularity source supplying safe defaults and rescuing cold users; a following or subscription source surfacing items from creators the user explicitly chose; and often a repeat source, things the user returns to.

Each source returns its own small list; the union, deduplicated, a few hundred strong, goes to the ranker, which does not care where a candidate came from.

The committee design has a second virtue: it is how product intent enters the system. Want more discovery? Widen the exploration source's slice. New-creator programme? Add a source. The retrieval mix is where a recommender's editorial character is set, one reason the diversity of the final page is mostly decided before the ranker ever runs.

Once the candidate set is a few hundred, the economics invert: features that were unaffordable across millions become routine, and the ranking stage's power comes precisely from this indulgence.

A production ranker's feature set typically spans four families:

• User features: long-term taste summaries, demographics where used, device, time of day.
• Item features: category, age, quality signals, historical engagement rates.
• Interaction features, the expensive gold: has this user seen this creator before, how did they respond, how similar is this item to the last ten things they finished, did they skip something like it yesterday.
• Context features: the surface itself, homepage rows tolerate exploration, autoplay does not, the session so far, what sits above in the feed.

Interaction features are the reason the two-stage split exists at all: they can only be computed for a specific user-item pair, which is affordable for hundreds of pairs and unthinkable for millions.

The output layer is a prediction, or several, of what the user will do with the item, click, finish, like, return tomorrow, and turning several predictions into one ordering is its own can of worms: lesson three is entirely about what the ranker should be predicting in the first place.

The ranker's ordered list is still not what the user sees. A final, mostly rule-based stage reshapes it, and the reshaping is not cosmetic:

• Diversity rules break up monoculture: not five items from the same creator in a row, not a whole page of one category, however high the scores. A ranked list optimal item-by-item is often a bad page, because the marginal value of the fourth similar item is near zero, a property pointwise scores cannot see.
• Business and policy constraints: licensing windows, age gates, regional availability, sponsored placements with their own contracted slots.
• Deduplication against recent exposure: the user saw this item three times already and did not bite; showing it a fourth time spends a slot on a known no.
• Explanation slots: because you watched X rows, which constrain their members by construction.

Architecturally this stage is cheap and fast-moving, product rules change weekly, models retrain on longer cycles, and it is the honest answer to a common confusion: when a recommendation looks strange, the cause is as often a re-ranking rule or a constraint as the model. Debugging a recommender starts with knowing which of the three layers, retrieval, ranking, or policy, made the decision in question.

The two-stage-plus-policy frame is a diagnostic instrument, and it transfers to every system you will meet.

When a platform recommends something baffling, decompose it: did retrieval surface it, meaning some source considered it plausible, perhaps a co-occurrence fluke or a popularity injection? Did ranking promote it, meaning the features genuinely scored it for this user? Or did policy place it, a sponsored slot, a diversity quota, a new-item boost?

When recommendations feel stale and repetitive, the suspect is usually retrieval: the candidate sources keep proposing the same neighbourhood, and no ranker can order its way out of a homogeneous candidate set.

When quality differs wildly between surfaces of one product, homepage good, autoplay poor, remember each surface typically has its own ranker and its own context features, sharing only retrieval.

And when a small product asks whether it needs all this: usually not at first. A catalogue of ten thousand items can be scored exhaustively by a single model within budget; the funnel earns its complexity somewhere past the point where that arithmetic breaks, and building it early buys operational cost without quality.

What the funnel does not explain is where those embedding-similarity candidates come from, and that is the next lesson: the two-tower model and the geometry of taste.