What a Hallucination Is, and Why It Happens

A language model is trained to predict the next token given the preceding ones. Nothing in that objective represents truth. What it represents is what text tends to follow what text.

So when a model produces a citation that does not exist, it has not malfunctioned. It has produced a plausible continuation, which is precisely what it was trained to do. A citation-shaped string in a citation-shaped context is exactly what the objective rewards, and whether the paper exists is not a property the training signal contained.

This reframing matters because it changes what kind of problem this is. Hallucination is not an error state the system enters, distinguishable from its normal operation. It is the same process producing the same kind of output, and the only difference is whether the result happens to correspond to the world.

Which has an uncomfortable consequence explored throughout this cursus: the model cannot tell you which of its outputs are hallucinated, because from inside the process there is no difference. Every mitigation in the next two lessons is therefore external, comparing output against something the model did not produce.

The word hallucination is also imperfect. It implies a perceptual malfunction, when the accurate description is confident generation under uncertainty.

The research literature converges on a dual-axis taxonomy, and using it prevents the most common category error in production debugging.

The first axis is intrinsic versus extrinsic. An intrinsic hallucination contradicts the input the model was given. An extrinsic hallucination cannot be verified from the input, because the input did not contain the information.

The second axis is factuality versus faithfulness. Factuality is correspondence with the world, whether the claim is true. Faithfulness is correspondence with the provided source, whether the output reflects what the input actually said.

The two come apart, and the direction that surprises people is a faithful output that is false. A retrieval system summarising an outdated document produces a summary that accurately reflects its source and misinforms the reader. Nothing in the pipeline malfunctioned.

The practical value of the taxonomy is diagnostic. A faithfulness failure is a generation problem: the model departed from material it was given, and better grounding or verification will help. A factuality failure with perfect faithfulness is a corpus problem, and no amount of grounding work fixes it, because the pipeline is working correctly on wrong input.

Teams routinely apply generation-side fixes to corpus-side problems and are puzzled when nothing improves.

Placing a failure in the right quadrant tells you which team owns the fix.

Unfaithful and false: the model departed from its sources and got it wrong. Generation-side, and the most tractable case, since verification against the retrieved passages catches it.

Unfaithful and true: the model departed from its sources and happened to be right, drawing on parametric knowledge. Often celebrated and it should not be, because the same behaviour produces the previous quadrant on a different input, and you have no control over which.

Faithful and false: the sources were wrong, outdated or contradictory. Corpus-side. Staleness governance from the company brain cursus, not a generation fix.

Faithful and true: the working case.

The second quadrant is the one to watch. A system whose accuracy comes partly from the model overriding its sources is unpredictable in exactly the way a regulated deployment cannot tolerate.

The property that makes hallucination dangerous rather than merely annoying is that a false output is indistinguishable, on its surface, from a true one.

In most information channels humans have learned to read confidence as weak evidence of reliability. A person who is uncertain hedges, hesitates, or qualifies. That heuristic is deeply installed and it is worthless here, because a language model's fluency reflects how well-formed the continuation is, not how well-supported.

A fabricated legal citation has the same crispness as a real one. An invented statistic has the same number of significant figures. A made-up API method has plausible naming and plausible arguments. There is no tell, because producing a convincing surface form is the thing the model is good at.

This is why hallucination resists the mitigation people reach for first, which is asking humans to check. Review catches errors that look wrong. Hallucinations do not look wrong, and a reviewer scanning for something that seems off will pass them.

It also explains why hallucination is worse in domains the reviewer does not know well, which is precisely where these systems are most often deployed for leverage. The failure mode and the use case are correlated.

Hallucination is not uniformly distributed. Several conditions raise the rate predictably, and knowing them tells you where to spend verification effort.

Sparse training coverage. Questions about obscure entities, recent events, or niche technical details sit where the model has little to draw on, and generation under thin evidence is where fabrication lives.

Specific formats with strong priors. Citations, statistics, dates, identifiers, version numbers, legal references. The model knows the shape well and the content poorly, which is the worst combination.

Long outputs. Error accumulates across a generation, and later claims are conditioned on earlier ones, so a small early invention can be elaborated confidently.

Pressure to answer. A model prompted in a way that makes refusal feel unavailable will produce something. Instruction-tuned models are trained toward helpfulness, and helpfulness under uncertainty looks like fabrication.

Conflicting or absent context. Where retrieval returns nothing relevant, or returns contradictory passages, the model bridges the gap.

And questions with false premises. Asked to explain something that does not exist, a model will frequently explain it rather than reject the premise.

The practical use: these conditions are detectable in advance, which means verification can be targeted rather than uniform, and targeting is what makes verification affordable.

A strand of theoretical work argues that hallucination cannot be fully eliminated from language models as they are currently constructed, and it is worth taking seriously rather than treating as pessimism.

The argument, developed by Xu and colleagues in 2024 under the title Hallucination is Inevitable, runs roughly as follows. A model is a computable function producing outputs over a formal system. There exist truths the model cannot produce and falsehoods it cannot avoid, for reasons analogous to classical undecidability results. So for any language model, there are inputs on which it will hallucinate.

What that does and does not establish is important.

It does not say hallucination rates cannot be reduced. They can, substantially, and the next two lessons are about how.

It does say that a design premised on eliminating hallucination is premised on something unavailable, and that a residual rate must be assumed rather than engineered away.

The operational consequence is a shift in what you build toward. Not a system that does not hallucinate, but a system where hallucination is unlikely, detectable when it occurs, contained when detected, and where the consequence of an undetected one is survivable.

That is a different specification, and it is achievable.

Four interventions organisations reach for first, and why each disappoints.

Telling the model not to hallucinate. Adding do not make anything up to a system prompt produces a model that asserts it is not making anything up. The instruction cannot be followed, because following it would require the model to distinguish supported from unsupported claims, which is the capability at issue.

Asking the model whether it is confident. Verbal confidence is generated by the same process as the content, so it reflects how confident such a statement usually sounds in this context rather than an internal state. Research on whether stated confidence tracks actual reliability finds the correspondence weak, and it can be actively misleading.

Using a bigger model. Larger models hallucinate less on many benchmarks and hallucinate more convincingly when they do. The base rate improves; the detectability does not, and for a reviewer the second matters more.

Adding a self-review step in the same conversation. Asking a model to check its own previous answer within the same context is weak, because the earlier claim is now in context and conditions the review toward consistency. Genuine self-checking requires an independent pass without the original output in context, which is the difference between a technique that works and one that reassures.

The pattern: any fix relying on the model's own judgement about its own output inherits the problem it is meant to solve.

Before building mitigations, decide what you are trying to achieve, because hallucination rate alone is a poor objective.

The rate is task-dependent to the point where a single number is meaningless. A model summarising a provided document and a model answering open questions from parametric knowledge are doing different things with different exposures, and averaging them tells you nothing about either.

The cost of an error varies enormously within one system. A fabricated detail in a brainstorming list costs nothing. A fabricated dosage costs a great deal. So the target should be stated per claim type, not per system.

And the relevant quantity is usually not the hallucination rate but the rate of hallucinations that reach a consequence. A fabrication caught by verification, or one that leads to an abstention, has been handled. That reframing is what makes the problem tractable: you are engineering the pipeline's failure rate, not the model's.

A workable target statement names the claim types that matter, the detection mechanism for each, the acceptable residual rate, and what happens when detection fires. For example: numeric claims must be traceable to a retrieved source or the system abstains, and untraceable numeric claims must occur in under one percent of responses, measured monthly on a fixed evaluation set.

That is specific enough to build against and to test, which the next two lessons do.