AnyLearn
All lessons
AIintermediate

Feedback Loops: The Model Trains on Clicks It Caused

A deployed recommender chooses its own future training data: it shows items, users respond to what was shown, and those responses become the next model's ground truth. This lesson maps the loop's consequences, exposure bias, popularity compounding, narrowing candidate pools, explains why offline metrics reward imitation of the loop, and covers the exploration budget that keeps the system learning.

Updated · AI-authored, review-gated · how lessons are made

recommender-systemsfeedback-loopsexplorationexposure-biasmachine-learningevaluation
Part of:Recommender Systems: The Two-Stage Machine
Not signed in: your progress and quiz score won't be saved.
Progress1 / 7

The loop, stated plainly

Every lesson so far treated the training data as a given: logs of users responding to items. The final complication is that in a deployed system, the logs are not given, they are chosen, by the previous version of the very model being trained.

The cycle runs continuously: the model selects what to show; users can only respond to what was shown; the responses are logged; the next model trains on the logs; the next model selects.

Key idea: a deployed recommender is not learning what users like. It is learning what users do with what it chose to show them, which is a different quantity, and the difference compounds with every cycle.

An item the system never surfaces generates no interactions, which reads, to the next training run, as evidence of no interest, which lowers its chance of being surfaced. The absence of evidence becomes evidence of absence, mechanically, with no one deciding it.

Nothing in this is exotic or malicious; it is what any learning system does when it controls its own data collection. But its consequences are the difference between recommenders on paper and recommenders in production, and managing the loop, rather than pretending it away, is the closing skill of this course.

Full lesson text

All 7 steps on one page, for reading, reference, and search.

Show

1. The loop, stated plainly

Every lesson so far treated the training data as a given: logs of users responding to items. The final complication is that in a deployed system, the logs are not given, they are chosen, by the previous version of the very model being trained.

The cycle runs continuously: the model selects what to show; users can only respond to what was shown; the responses are logged; the next model trains on the logs; the next model selects.

Key idea: a deployed recommender is not learning what users like. It is learning what users do with what it chose to show them, which is a different quantity, and the difference compounds with every cycle.

An item the system never surfaces generates no interactions, which reads, to the next training run, as evidence of no interest, which lowers its chance of being surfaced. The absence of evidence becomes evidence of absence, mechanically, with no one deciding it.

Nothing in this is exotic or malicious; it is what any learning system does when it controls its own data collection. But its consequences are the difference between recommenders on paper and recommenders in production, and managing the loop, rather than pretending it away, is the closing skill of this course.

2. Where the loop bites, mechanism by mechanism

The single cycle produces several named pathologies, each a different place the same arrow lands.

Exposure bias is the foundation: interaction data exists only for shown items, so all learned preferences are conditioned on past exposure decisions. Every other effect is this one compounding somewhere specific.

Popularity compounding: popular items get shown, gathering more interactions, sharpening the model's confidence in them, earning more exposure. Head items entrench; equally good tail items starve on no evidence. The rich-get-richer shape is measurable in most production systems as a popularity skew far beyond organic preference.

User narrowing: the same dynamic per person. Engaging with one topic concentrates future recommendations there, harvesting more same-topic engagement. The user's observed profile narrows faster than their actual taste, because the menu narrowed first. Filter-bubble discussion in the media is this mechanism, wearing its social consequences.

Candidate pool decay: retrieval sources trained on interactions propose from the interacted region, so the ranker's whole menu drifts toward the already-shown, and the funnel's earlier stages quietly enforce the loop before ranking gets a vote.

And model confidence inversion: the system is most certain exactly where it has looked most, and knows least about the items and users it has been neglecting, the opposite of where curiosity should point.

flowchart TD
A["Model ranks and shows items"] --> B["Users respond to what was shown"]
B --> C["Logs record the responses"]
C --> D["Next model trains on logs"]
D --> A
C --> E["Unshown items: zero evidence"]
E --> F["Read as zero interest"]
F --> A

3. Why offline evaluation cannot see any of this

The loop also corrupts the measuring instruments, which is why it survives in systems run by careful people.

Standard offline evaluation holds out a slice of the logged interactions and asks the new model to predict them. But the held-out interactions are themselves products of the loop: they happened on items the old system chose to show. A new model that imitates the old system's exposure pattern predicts that test set beautifully; a model that would explore differently, surfacing items the old system never showed, gets no credit for any of it, because the log contains no evidence about those items by construction.

Predict first

Two candidate models: one closely mimics the current production ranking, one substantially diversifies exposure. Offline evaluation on logged data says the mimic is clearly better. What does that result actually establish?

The practical consequence is a hierarchy of trust: offline evaluation for cheap filtering of obviously bad candidates, randomised online experiments for truth, and standing scepticism toward any large offline win, which is as likely to be loop-imitation as improvement.

4. Exploration: paying for information on purpose

The loop starves the system of evidence about the unshown; the counter-measure is to buy that evidence deliberately. Exploration means occasionally showing items the current model would not have chosen, accepting a small immediate engagement cost in exchange for data the exploit-only policy can never collect.

The machinery, from crude to principled:

  • Random injection: a small slice of slots gets near-random or freshness-pool items. Blunt, but it produces the gold-standard unbiased data that propensity methods and honest evaluation need.
  • Epsilon-greedy: with small probability, swap a chosen item for a candidate the model is unsure about. Simple to implement and reason about.
  • Uncertainty-directed exploration, the bandit family: prefer to explore where the model's uncertainty is highest, upper-confidence-bound methods and Thompson sampling being the classic shapes. More information per unit of sacrificed engagement, at the cost of needing uncertainty estimates at all.
  • Structural exploration: dedicated surfaces where discovery is the point, new-release rows, discovery playlists, so exploration happens where users expect novelty rather than diluting high-intent surfaces.

Gotcha: exploration is unpopular in the metrics review, because its cost is visible this week, slightly lower engagement on explored slots, while its benefit, a model that still knows things next quarter, appears nowhere in the dashboard. Teams that cut exploration in a metrics push are borrowing engagement from the future; the debt arrives as a stale catalogue and an entrenched head.

5. Correcting the log: propensity weighting

Between blunt randomisation and full online experiments sits the statistical repair kit: use the log, but reweight it to undo the exposure policy's thumb on the scale.

The tool is inverse propensity scoring. If the system showed item a to context x with probability mu, and a new policy pi would have shown it with probability pi, then each logged interaction is reweighted by the ratio:

V^(π)=1n∑i=1nπ(ai∣xi)μ(ai∣xi) ri\hat{V}(\pi) = \frac{1}{n} \sum_{i=1}^{n} \frac{\pi(a_i \mid x_i)}{\mu(a_i \mid x_i)} \, r_i

Interactions the old system rarely allowed to happen, low mu, get up-weighted, compensating for their under-representation; interactions the old system produced constantly get discounted. Under the right conditions this yields an unbiased estimate of how the new policy would perform, from data the old policy collected.

The fine print is where practice lives:

  • The weights need logged propensities, which is why recording mu at impression time is this course's most repeated advice.
  • Variance explodes where pi and mu disagree most. A rare exposure carrying a huge weight makes the estimate swing on a handful of events, precisely in the interesting region. Weight clipping and self-normalised variants trade a little bias back for stability.
  • Zero-probability exposures cannot be repaired. If the old system never showed an item, no reweighting conjures its missing evidence, which is why propensity methods complement exploration rather than replacing it.

6. Managing the ecosystem, not just the user

A recommender at platform scale is not choosing items for users; it is allocating attention across a supply side that adapts, and the loop runs through the suppliers too.

Creators, sellers and artists observe what the system rewards and produce more of it: the objective function of the ranker becomes, within months, a design brief for the catalogue. Optimise strong openings and creators front-load; reward posting frequency and quality fragments into volume. This supplier loop is faster and more deliberate than the user loop, because suppliers are actively reverse-engineering the system.

Meanwhile the cold-start exposure problem from earlier lessons is, at ecosystem scale, a market-design question: which new items get the scarce exploration slots decides which creators survive, and a platform whose loop entrenches incumbents ages visibly, its catalogue calcifying while competitors with fresher loops feel alive.

The levers here are policy as much as modelling: minimum-exposure guarantees for new items, so every entrant gets a measured audition rather than zero; exploration budgets earmarked by segment, new creators, under-served topics; anti-compounding terms that damp pure popularity in retrieval sources; and supply-side dashboards, exposure concentration, survival rates of new entrants, watched with the same seriousness as engagement, because they are the leading indicators of the catalogue the users will experience next year.

None of these choices is neutral, and that is the honest framing: an attention-allocation policy exists whether or not anyone wrote it down. The written version is merely the one that was chosen.

7. The operating checklist

Compressing the course's final lesson into the habits that distinguish teams who run the loop from teams the loop runs:

  1. Log propensities. Record, for every impression, the probability with which the system chose it. Without propensities, no debiasing method and no honest off-policy estimate is ever possible retroactively; it is the cheapest data you will ever wish you had collected.
  2. Keep a permanent random slice. Even a fraction of a percent of randomised exposure provides the unbiased measurement floor everything else calibrates against.
  3. Fund exploration as a budget line, with an owner, immune to quarterly engagement pushes, sized per surface. Structural exploration on discovery surfaces buys the most information per unit of user tolerance.
  4. Distrust large offline wins; demand online confirmation, and hold long-running holdbacks for ranking changes, the loop's damage is slow and invisible at experiment timescales.
  5. Watch supply-side health, exposure concentration, new-entrant survival, catalogue coverage of retrieval, as leading indicators of the experience two quarters out.
  6. Audit the loop's composition periodically: what did we actually show this month, versus what exists, the gap is the loop's current shape, and it drifts.

With that, the course's arc closes. The two-stage machine makes recommendation computable; embeddings make taste geometric; objectives encode what the product values; and the loop, managed or unmanaged, decides what the whole system becomes over time. The teams that do this well are not the ones with the cleverest models: they are the ones who never forgot that the system is inside the world it is trying to measure.

Check your understanding

The lesson ends with a 5-question quiz. Take it in the player above to see your score.

  1. What does a deployed recommender actually learn from its logs?
    • The true preference distribution of its users
    • What users do with what the previous model chose to show them, which conditions all learned preferences on past exposure
    • The intrinsic quality of each catalogue item
    • Nothing, since logs are discarded between retrains
  2. Why do popular items entrench under the loop even when equally good tail items exist?
    • Popular items have better metadata
    • Users explicitly prefer familiar items
    • The ranking model overfits to item age
    • Exposure generates interactions, which raise model confidence and earn more exposure, while unshown items starve on zero evidence
  3. A new model beats production by a wide margin in offline evaluation on logged data. What is the strongest alternative explanation to genuine improvement?
    • It resembles the system that generated the test data, and models that would explore differently cannot receive credit because the log lacks evidence about unshown items
    • The test set was too small
    • The baseline was under-trained
    • Offline metrics favour deeper networks
  4. Why does cutting the exploration budget look good this quarter and cost dearly later?
    • Exploration slots are the most expensive to serve
    • Its cost, slightly lower engagement on explored slots, is visible now, while its benefit, evidence that keeps future models informed, never appears in the current dashboard
    • Users complain about random recommendations immediately
    • Regulators require minimum exploration
  5. Why should every impression be logged with its propensity, the probability the system chose it?
    • Propensities compress the logs for cheaper storage
    • Serving latency improves when propensities are cached
    • Propensities enable debiasing and off-policy evaluation, and cannot be reconstructed retroactively if not logged
    • Advertisers require propensity reporting

Related lessons

AI
intermediate

Ranking and Objectives: What Should the Model Optimise?

The ranker is a prediction machine, and the hard question is what it should predict. Clicks are plentiful and poisonous, watch time bends toward length, likes are rare and unrepresentative. This lesson covers implicit feedback, the position bias baked into every training log, multi-objective ranking, and calibration.

7 steps·~11 min
AI
intermediate

The Two-Stage Machine: Why No Model Ranks the Whole Catalogue

A recommender has milliseconds to pick ten items from millions, and no model good enough to rank them all is cheap enough to run on them all. The industry's answer is a funnel: cheap candidate generation cuts millions to hundreds, an expensive ranker orders those hundreds. This lesson builds that architecture, its latency arithmetic, and the multi-source retrieval layer real systems run.

7 steps·~11 min
AI
advanced

Evaluating a Book Model Honestly

If your cost per round trip equals the move you are trying to capture, you need 100 percent directional accuracy to break even. This lesson computes that hurdle, replaces accuracy with metrics tied to a tradeable decision, and covers the capacity and latency limits that decide whether a real edge is worth anything.

10 steps·~15 min
AI
advanced

Why Reported Order Book Results Do Not Replicate

At a one-event horizon, 92 percent of mid-price labels are exactly no-change, so a model that always predicts flat scores 92 percent accuracy. This lesson computes that baseline across horizons and works through the four mechanisms that turn a genuine measurement into a number nobody can reproduce.

10 steps·~15 min