Two Towers: How Taste Becomes Geometry

The deepest idea in recommendation predates deep learning entirely: you do not need to understand items to recommend them, because behaviour already encodes similarity.

If many of the people who watched A also watched B, then A and B are related, whatever they are about. If your history overlaps heavily with another user's, their future is a decent forecast of yours. This is collaborative filtering: recommend from the collective pattern of behaviour, with no item descriptions, no categories, no content analysis at all.

Its power made the field: behavioural similarity captures affinities no metadata expresses, the two directors whose fans overlap despite different genres, the products bought together for reasons no taxonomy anticipated.

Its two weaknesses also made the field's agenda:

• Sparsity. Any one user touches a vanishing fraction of the catalogue; most user-item pairs carry no evidence.
• Novelty blindness. An item nobody has interacted with has no behavioural pattern, so pure collaboration cannot recommend it to anyone: the cold-start problem, which stalks every design in this lesson.

Everything since is machinery for compressing sparse behaviour into something dense enough to generalise, and the compression of choice is the embedding.

Picture the interaction data as a giant table: users as rows, items as columns, a cell marking that this user engaged with that item. Almost every cell is empty, and the filled cells are all you know.

Matrix factorization, the technique that famously dominated the Netflix Prize competition era, makes one austere assumption: this vast sparse table is secretly low-rank, meaning it can be approximated as the product of two thin matrices. One gives every user a vector of k numbers, one gives every item a vector of k numbers, and the model's prediction for any user-item pair is simply the dot product:

Training nudges these vectors so that dot products are high on observed interactions and not elsewhere; the vectors are learned from behaviour alone.

What emerges is remarkable: the k dimensions become axes of taste that nobody designed, one direction separating mainstream from niche, another tracking tone or genre, legible only through which items land where. Users and items live in the same space, and "this user would like that item" has become "these two points are aligned".

The limitation is equally instructive: each user's vector is a free parameter looked up by id. New user, no vector. New item, no vector. The cold start is built into the parameterisation itself, and fixing that is exactly what the modern version changes.

The two-tower model keeps matrix factorization's geometry and replaces its lookup tables with neural networks, which is a bigger upgrade than it sounds.

The user tower is a network that consumes user features, the sequence of recent interactions, long-term aggregates, context like device and hour, and outputs a user embedding. The item tower consumes item features, its category, creator, text and media-derived signals, engagement statistics, and outputs an item embedding in the same space. Relevance remains the dot product of the two outputs.

Because the towers compute embeddings from features rather than looking them up by id, generalisation changes character. A brand-new item gets a sensible embedding from its content features alone, near similar items, before its first interaction: cold start softened structurally. A user's embedding moves the moment their recent-history feature changes, no retraining required.

Training pairs the towers on logged interactions: embeddings of users and the items they engaged pull together, others push apart, with much engineering care in choosing the negatives, random catalogue items are too easy, items shown-but-skipped carry information but also position bias, a preview of the next lesson's territory.

The architecture has a striking discipline: the user tower never sees item features, the item tower never sees user features, and they meet only in a dot product. A model allowed to mix user and item features in shared layers would be strictly more expressive. Why forbid it?

This is also the cleanest illustration in the course of architecture serving systems constraints: the towers are shaped by where the computation must run, offline for items, online for the user, index in between. The expressive, feature-mixing models are not discarded; they are exactly what the ranking stage runs on the short-list, where per-pair passes are affordable. The funnel's two stages are two points on the same expressiveness-versus-scale trade.

Operationally, the two-tower system runs as three loops at three speeds, and knowing them is knowing where the failure modes live.

• The training loop, slowest. Towers retrain on recent interaction logs, daily or continuously, and each retrain shifts the embedding space itself. A subtle consequence: user and item embeddings are only comparable within the same model version, so index and user-tower deployments must be versioned together, or dot products become nonsense quietly.
• The item indexing loop. New and updated items flow through the item tower into the ANN index on a schedule of minutes to hours. The index rebuild-versus-incremental-insert trade, and the recall cost of approximation, are the vector-search machinery covered in the Vector Databases in Depth course; the recommender inherits all of it.
• The request loop, fastest. Per request: assemble user features, one user-tower inference, one ANN query, then onward to ranking. Feature assembly is usually the latency surprise here, gathering a user's recent history from storage often costs more than the model.

Gotcha: the freshness of the three loops differs by orders of magnitude, and users notice the seams. The classic symptom: an item you bought this morning is still being recommended tonight, because the interaction reached the logs but the retrieval layer's view of you refreshes on the slow loop. Real-time feature pipelines exist largely to close this gap for the signals users feel most.

The two-tower design softens cold start; it does not solve it, and the remaining hard cases deserve honest treatment because product decisions hang on them.

Cold items are the softened case. The item tower places a new item by its content features, near its siblings, and that placement is only as good as the content features are predictive. For a new track by a known artist, excellent. For a new creator's first upload with a generic title, the embedding is generic too, and the item needs the exploration machinery of the next lesson, deliberate exposure to gather the behavioural signal content could not provide.

Cold users are harder, because the user tower's most predictive features, the interaction history, are exactly what is missing. The standard ladder, in order of value: use whatever context exists, locale, device, referral source, signup answers; lean on popularity within that context, the honest fallback; harvest the first session's signals aggressively, the first three interactions cut the candidate space dramatically; and design onboarding to buy signal cheaply, pick three topics you like is a feature-acquisition device.

The cold platform, no users and no items with history, is the degenerate case where collaborative machinery has nothing to grip: early-stage products run content similarity and editorial curation until behaviour accumulates, and pretending otherwise just adds infrastructure to a problem it cannot help.

The general principle: cold start is not one problem but a spectrum of missing-signal problems, and the fix is always the same in shape, substitute the missing behavioural signal with the best available prior, then spend exploration to replace the prior with evidence.

A closing calibration on embedding-based retrieval, because both over-trust and under-trust are common.

What the geometry does well: it generalises across the catalogue from sparse evidence, surfaces non-obvious affinities that no metadata predicted, adapts to a user within a session through the feature pipeline, and serves all of it from millions of items in milliseconds. It is deservedly the workhorse.

What it structurally cannot do: it can only interpolate within observed behaviour. The dot product finds items aligned with where the user already is; it has no mechanism for judging whether the user would love something orthogonal to their history, because no signal placed that item near them. Left alone, embedding retrieval is a machine for giving people more of what they have already chosen, with all the self-reinforcement that implies, the full treatment of which, feedback loops, exposure bias, and the exploration that counters them, is the final lesson's subject.

And a modelling honesty note: the dot product is a similarity, not a preference strength. High alignment means the item resembles the user's history pattern, which correlates with engagement but is not the same as it, items can be similar and unwanted, novel and wanted. That gap between geometric plausibility and actual preference is precisely the job of the ranking stage, with its richer features and its trained objectives, and choosing those objectives well turns out to be the hardest problem in the whole pipeline.