The Average Trap, Model Collapse, and Responsible Practice

The previous lesson ended with a checklist: report your configurations, preregister, validate against humans. Follow it fully and you address the flexibility problem.

Two failure modes remain, and they are structural rather than procedural. They operate even when every methodological box is ticked, because they follow from what the technology does rather than from how carefully it is used.

The first concerns what these models predict, and pulls research toward the typical case. The second concerns what happens when synthetic output re-enters the evidence base, and degrades it over time.

Both are argued in "The GenAI Future of Consumer Research" by Ming-Hui Huang and Roland Rust, in the Journal of Consumer Research, volume 52, issue 1, 2025. It is a conceptual piece rather than an experimental one, so treat it as a well-argued framework rather than a measured result. The two mechanisms it names have become the standard vocabulary for these risks.

Huang and Rust's first mechanism, which they name the average trap, follows directly from the training objective.

A generative model predicts the most likely continuation. Applied to consumer behaviour, that means predicting the most likely response, which is to say the response of the average consumer. Research built on that instrument inherits its target: it tends to predict and explain average behaviour, and the tendency is structural rather than a tuning artefact.

Their applied consequence is the sharp one. Marketing exists to differentiate, and an instrument biased toward the generic produces recommendations that fail to distinguish one brand from another. If every competitor researches with the same models trained on overlapping corpora, they converge on similar answers about what the typical customer wants. The research method itself becomes a homogenising force.

The authors' prescription follows: consumer research must stay human-centric, deliberately preserving human differences rather than smoothing them away, and needs methods that detect when insights are drifting toward the bland or the non-human.

The average trap sounds abstract until you notice how much of marketing lives in the distribution's tails.

Early adopters are by definition atypical, and they are who new categories are launched to. Heavy users generate disproportionate revenue in most categories and behave unlike the median. Defectors are the minority whose reasons you most need. Emerging preferences start as fringe positions and are indistinguishable from noise at the point where knowing about them is valuable.

An instrument that reliably reports the centre and compresses the edges is therefore weakest precisely where the commercially interesting answers are. And it fails silently: it returns a confident, articulate description of the typical customer, which reads like an insight rather than an absence.

Note how this compounds the variance compression from the second lesson. That was a statistical property of the sampling process. This is a property of the prediction objective. They push the same direction, which is why synthetic consumer research skews toward the reassuring and the already-known.

The second mechanism concerns what happens over time, and it has a firm empirical basis outside marketing.

Model collapse was demonstrated by Ilia Shumailov and colleagues in "AI models collapse when trained on recursively generated data", published in Nature in 2024. Train a generative model on data produced by a previous generation of models, repeatedly, and quality degrades in a specific pattern.

The paper distinguishes two stages. In early collapse, distributional errors accumulate and the model drifts from the true distribution. In late collapse, low-frequency events disappear permanently. Within a few generations, original content is replaced by unrelated nonsense.

Notice which part dies first: the tails. The rare events go, exactly the atypical consumers the average trap already underweights.

Huang and Rust apply this to the research pipeline. As synthetic data proliferates and is mistaken for human data, consumer research risks shifting toward predicting and explaining machine behaviour rather than human behaviour, in patterns that no longer make human sense.

The loop is what makes this different from an ordinary measurement error. Synthetic findings get published, published text becomes training data, and future models learn from their predecessors' output. Each pass strips more of the tails, and no participant in the loop is doing anything obviously wrong.

The literature converges on a set of practices. Drawing on the guidance from Sarstedt and colleagues and the reproducibility work already covered, six rules do most of the work.

Validate in your domain, or do not claim generalisation. Fidelity demonstrated elsewhere transfers to nothing.

Keep a human anchor. Calibrating synthetic responses against a smaller human sample is the best-supported use, and it preserves a measurable error estimate.

Report the multiverse. Model version, temperature, prompts, and results across configurations, not just the reported one.

Treat it as a hypothesis generator by default. Promote a finding to a conclusion only after human confirmation.

Never use it where the tails are the question. Segmentation, polarisation, early adopters, and niche appeal are the documented weak spots.

Label synthetic data permanently. This is the only defence against the contamination loop, and it only works if everyone does it. Data that leaves your organisation unlabelled becomes someone else's ground truth.

Synthetic respondent products are now sold commercially, and the marketing outpaces the evidence. A short translation guide.

One question separates serious offerings from the rest: what does it get wrong, and how do you know? A vendor with real validation data can answer specifically, naming categories where correspondence is poor. A vendor who cannot has not measured it.

The version issue deserves emphasis because it is easy to miss. If a provider silently updates the underlying model, a tracking study comparing this quarter to last quarter is comparing two different instruments, and any movement you observe may be the instrument rather than the market.

Four claims, separated from the enthusiasm and the backlash alike.

The mechanism is real. Models conditioned on demographic backstories produce responses that track human subgroups to a measurable degree, and reproduce qualitative economic structure such as downward-sloping demand without being asked. Argyle's algorithmic fidelity is a genuine finding.

The accuracy is domain-specific and often poor. Result patterns vary considerably across domains, magnitude estimates are frequently unreliable, and wrong-signed cases are documented in the marketing literature.

Single-configuration results are close to uninterpretable. Defensible setup choices moved human-to-silicon correlations from .23 to .84 on one published study, so any result reported from one configuration is a draw from a wide unknown distribution.

Two failure modes survive good methodology. The average trap follows from the prediction objective; model collapse follows from synthetic data re-entering the corpus. Neither is fixed by careful use.

The defensible position is narrow and useful: silicon samples are an input to human research, never a replacement for it, and any claim otherwise needs domain-specific validation and a reported multiverse before it deserves weight.