Model Collapse, and the Rule That Avoids It

Take a model, generate data with it, train the next model on that data, and repeat. Shumailov and colleagues showed in Nature in 2024 that this loop degrades the model, and the degradation has two distinguishable stages.

Early collapse is a drift. Distributional errors accumulate, and the model's output distribution moves away from the true one. The text still reads well. What is disappearing is accuracy about how often things occur.

Late collapse is a loss. Low-frequency events vanish from the distribution entirely, and the model converges toward a narrow region of high-probability output. The paper describes the resulting defects as irreversible: the tails of the original distribution are gone.

The mechanism is not exotic. Each generation samples finitely from its parent, and rare events are exactly the ones a finite sample is likely to miss. Once missed, they are absent from the child's training data, so the child assigns them lower probability, so the grandchild is even less likely to sample them.

Errors compound in one direction, toward the mode.

The collapse result is real and it is conditional, and the condition sits in the experimental setup rather than in the headline.

In the setup that collapses, each generation's synthetic data replaces the previous data. Generation two trains on generation one's output. Generation three trains on generation two's output. The original real data is present at the start and gone thereafter.

Gerstgrasser and colleagues asked what happens under the other assumption: that data accumulates. Generation three trains on the real data plus generation one's output plus generation two's output.

The difference is decisive. They confirm that replacing real data with each generation's synthetic data does tend toward collapse, and then show that accumulating successive generations alongside the original real data avoids it, across a range of model sizes, architectures and hyperparameters.

The analytical result is the sharpest statement of it. Under accumulation, test error has a finite upper bound. Under replacement, it grows without bound.

That is a qualitative difference, not a matter of degree, and it yields a rule short enough to remember: accumulate, do not replace.

The intuition behind the analytical result is worth holding, because it tells you what the real data is actually doing.

Under replacement, each generation's only evidence about the world is the previous generation's output. Any error introduced at one step is treated as ground truth at the next, and there is nothing to contradict it. The chain is a random walk with no anchor, so deviations accumulate and the distance from truth grows without limit.

Under accumulation, the original real data is present in every generation's training set. Whatever a synthetic generation gets wrong, the real data still says otherwise. The real portion acts as an anchor, and each generation's error is corrected against it rather than inherited.

The real data's job, then, is not to be a majority of the tokens. It is to be present at all, permanently. It is what stops a drift from becoming a trajectory.

The practical translation is a policy about datasets rather than a hyperparameter. Never delete the real data. Never train a generation exclusively on the previous generation's output. Both are obvious once stated and both are violated routinely, usually for storage reasons or because a pipeline was written to overwrite.

The distinction is entirely structural, and it fits in a diagram.

On the replacement path, real data trains generation one. Generation one produces synthetic data, which becomes the training set for generation two, and the real data is dropped. Each subsequent generation sees only its parent's output. Error grows without bound, and the tails disappear permanently.

On the accumulation path, the real data never leaves the training set. Each generation's synthetic output is added to a pool that already contains the real data and every previous generation's contribution. Error stays bounded.

The operational reading is that the arrow removing the real data is the whole difference. It is one line in a data pipeline, usually written for a reason that has nothing to do with model quality.

And note what is not on the diagram. Nothing here depends on the quality of the synthetic data or on the filters from the previous lesson. Good filtering slows collapse under replacement and does not prevent it, because the anchor is missing regardless of how clean the drift is.

Collapse destroys the tails first, and it is worth being concrete about why that is the expensive part rather than a technicality.

The tail of a distribution is where the unusual cases live. Rare failure modes. Minority dialects and phrasings. Uncommon but valid formats. Domain-specific vocabulary that appears in one document in ten thousand. Legitimate inputs that look like mistakes.

A model that has lost its tails performs identically on the common cases, which is most of any test set, and fails on the uncommon ones. Aggregate metrics barely move. The failures land on whoever is unusual.

That is also why the damage is hard to detect with the evaluation people naturally run. Average accuracy over a representative sample weights by frequency, and the tail is by definition rare in a representative sample.

The detection method that works is to evaluate on a deliberately non-representative set, oversampling exactly the cases you expect to be rare, and to track them as a separate number rather than folding them into an average.

The catalogue's path on synthetic survey respondents makes the same point from the market research side, where the tail is minority subgroups rather than rare inputs.

Given both real and synthetic data, the question is how much of each, and the answer is less delicate than it first appears.

The first thing to know is that the ratio is a weaker lever than the presence of real data at all. The accumulation result says the anchor prevents unbounded error; it does not say the anchor has to be half the corpus.

A workable default is to lean synthetic where the synthetic data addresses a coverage gap, and keep a real fraction that is small but never zero. The synthetic portion supplies breadth; the real portion supplies correction.

What governs the choice is what each source is for. If synthetic data is filling in rare cases, its share can be large, because the real data is not covering them anyway. If synthetic data is standing in for the main distribution, keep the real share high, because the main distribution is exactly what the real data describes accurately.

And weight rather than duplicate. Oversampling a small real set by repeating it is not equivalent to having more real data, and repeated examples risk memorisation.

Then measure, because the ratio interacts with the task and the model in ways nothing published will settle for your case.

If a single sentence from this path survives, it should be this one. Never evaluate on synthetic data.

The reason is that a synthetic evaluation of a synthetically trained model measures agreement between two outputs of the same process. It will report a high number, and that number is a measure of self-consistency rather than of capability.

The failure is subtle because the evaluation set looks legitimate. It was generated separately, it was not in the training set, it passed the contamination check. And it was produced by the same generator with the same blind spots, so anything the generator misunderstands is misunderstood identically in both the training data and the test.

The requirement is real held-out data, collected independently of the generation pipeline, ideally after the generator's training cutoff.

This holds even when real data is scarce, which is usually the reason people went synthetic in the first place. A hundred real examples used purely for evaluation are worth more than ten thousand synthetic ones, because they are the only thing in the system capable of telling you the pipeline is wrong.

Spend scarce real data on measurement before spending it on training.

The literature discusses collapse as a deliberate experimental setup. In practice the loop closes by accident, and the accidental version is harder to see.

A model is deployed. Its outputs are published, or logged, or used to draft content a human lightly edits. Those outputs re-enter the corpus. Later training runs collect data that now includes the model's own output, unlabelled and indistinguishable from human text.

Nothing about this is a decision anyone made. It is what happens when generated text becomes part of the environment.

Inside a single organisation the same pattern appears in miniature. A model drafts support responses. Agents approve most of them with minor edits. The approved responses become next quarter's training data. The loop is closed, and it looks like a virtuous cycle of real human-approved data.

The defence is provenance, which is why the previous lesson insisted on recording it. If you know which examples were model-generated, you can decide how to weight them. If that field was never written, the distinction is unrecoverable, and every later dataset is of unknown composition.

Human-in-the-loop is the usual answer to everything in this lesson, and it is worth being precise about which problems it solves, because it does not solve all of them.

It does break the recursion, provided the human is genuinely evaluating rather than approving. A reviewer who reads the model's answer and decides whether it is correct is injecting information from outside the loop, which is the anchoring role real data plays.

It does not break the recursion when the human is rubber-stamping. Approval rates in the high nineties on any non-trivial task usually indicate the reviewer is checking plausibility rather than correctness, and plausibility is exactly what the generator optimises. Under that regime the human is a formatting filter with a salary.

And it does not fix coverage. A reviewer can only evaluate the examples the generator produced, so the regions the generator never visits stay empty no matter how careful the review.

The efficient use of review effort is therefore not uniform sampling. It is targeting the cases the automatic filters were least certain about, and periodically auditing a random slice to check the filters are still calibrated.

Collecting the whole path into what survives contact with a real project.

Name where the information comes from before generating anything. A teacher, a verifier, a template, or real inputs. If the answer is the model you are about to train, the loop is circular.

Spend the effort on conditioning rather than temperature, because coverage is decided by what you condition on and is the property that decides whether a student generalises.

Generate many and keep few, with the strongest available checker. Machine verification where possible, self-consistency where the answer is determinate, a calibrated judge otherwise.

Deduplicate on meaning, not bytes, and check contamination against every evaluation set you use.

Accumulate, never replace. The real data stays in the mix permanently, in whatever proportion, because its job is to be an anchor rather than a majority.

Record provenance on every example, because the accidental loop is invisible without it.

And evaluate only on real held-out data. Every other rule here degrades a model. Breaking this one means you will not find out.