Why Fluent Text Defeats Your Judgement

People assess claims partly on content and substantially on cues that correlate with reliability. Those cues were never perfect and they were useful, and generated text breaks most of them at once.

What we historically read as signals of expertise. Fluency, because writing clearly about something usually required understanding it. Confidence, because people who know a subject tend to hedge less. Structure, because organising an argument well demanded having one. Vocabulary, because using terms correctly implied familiarity. And detail, because specifics are harder to produce than generalities.

Every one of those correlated with competence because producing them was costly. That is why they worked.

Generated text supplies all five at essentially zero cost, and it supplies them uniformly, decoupled from whether the underlying content is correct. A model writing about something it represents poorly produces text as fluent, confident, structured and specific as text about something it represents well.

So the cues have not merely become weaker. They have become uninformative, while continuing to feel informative, which is the dangerous combination.

That is the same structure as the security cursus in this catalogue, where the bad-grammar heuristic stopped tracking anything while people kept applying it. Here the effect is broader, because the cues are not a learned checklist but how everyone has always evaluated writing.

The rest of this lesson covers four specific mechanisms by which this plays out, and the next covers what to replace the cues with.

The first mechanism is that the ease of reading something influences whether you believe it, independently of content.

This is well studied under the name processing fluency. Work by Norbert Schwarz, Rolf Reber, Adam Alter, Daniel Oppenheimer and others has repeatedly found that statements which are easier to process are judged more likely to be true, more likely to be familiar, and produced by more competent authors.

The effects are demonstrable with manipulations that have nothing to do with meaning. A statement in a clearer font is rated more truthful than the same statement in a degraded one. A rhyming aphorism is rated more accurate than a non-rhyming paraphrase with identical content. A name that is easier to pronounce is judged more trustworthy.

These are small effects individually, and they are systematic, they operate below awareness, and knowing about them does not switch them off.

Why this matters here specifically. Generated prose is optimised for exactly the property that drives the effect. It is smooth, well-structured, familiar in register, and free of the friction that comes with a person working out what they think as they write. It is the most fluent text most people encounter.

So it collects a truth bonus purely from its form.

And note the asymmetry with a human expert, which is the practical consequence. A knowledgeable colleague explaining something difficult often speaks awkwardly: they hesitate, qualify, restart, and say it depends. Those are marks of genuine engagement with a hard question, and they read as less credible than a generated paragraph that has none of them.

The correction is not to distrust clear writing. It is to notice that you cannot use clarity as evidence any more.

The second mechanism predates these tools by decades and was studied in aviation, medicine and process control, where the findings transfer directly.

Automation bias is the tendency to over-rely on automated output: to accept a system's recommendation without adequate checking, and to fail to notice problems the system did not flag. Raja Parasuraman and Victor Riley's 1997 framework distinguished misuse, disuse and abuse of automation, and work by Linda Skitka and colleagues demonstrated both forms experimentally.

The two error types are worth separating because they need different countermeasures.

Commission errors: acting on a wrong recommendation that adequate checking would have caught. The operator had the information to detect the problem and deferred to the system instead.

Omission errors: failing to notice something because the system did not raise it. This is the more insidious one, because the operator is not aware of having made a decision at all. Absence of an alert is read as absence of a problem.

What the research also found, and it matters for how organisations respond. Automation bias is not eliminated by training people about it, is not confined to inexperienced operators, and increases with the system's general reliability. A system that is right most of the time trains you to stop checking, which is rational on average and catastrophic in the specific cases where it is wrong.

That last point deserves emphasis. The better a tool gets, the stronger the bias becomes. So improving accuracy does not straightforwardly improve outcomes, because it degrades the vigilance that catches the residual errors.

Which is why the countermeasures in the next lesson are structural. Deciding to be more careful does not work, and it has been tested.

Tracing the path from a generated claim to a decision, with the four failure points marked.

A claim arrives, fluent and confident. Processing fluency gives it an unearned credibility bonus at the moment of reading, before any evaluation begins.

You then decide whether to check. Automation bias pushes toward not checking, and it pushes harder the more reliable the tool has been. This is the highest-leverage point in the whole path, and it is a decision most people do not experience making.

If you do not check, the claim proceeds to the decision directly.

If you do check, you check against your own understanding, which is where the illusion of explanatory depth operates: a clear explanation makes you feel you understand the subject well enough to evaluate it, when what you have acquired is familiarity with the explanation.

And if you push back, sycophancy means the model may simply agree with you, which feels like confirmation and is not.

What the diagram is built to show. Only the second node is a real decision point, and it is the one people are least aware of. The first and third operate below awareness, and the fourth mimics the feedback that would normally correct you.

So the intervention has to be at the checking decision, made before the claim arrives rather than in response to it, which is what the next lesson builds.

The third mechanism concerns your assessment of your own understanding, which is what you use to evaluate anything.

Leonid Rozenblit and Frank Keil described the illusion of explanatory depth in Cognitive Science in 2002. People asked how well they understand an everyday device, a zip, a flush toilet, a bicycle, rate their understanding highly. Asked to write a step-by-step explanation of how it works, they discover they cannot, and their subsequent self-rating drops sharply.

The finding is that people mistake familiarity with a thing for understanding of it, and the illusion is specific to explanatory knowledge rather than to facts or procedures.

Why generated explanations amplify this precisely. Reading a clear explanation produces exactly the sensation the illusion is made of: everything follows, nothing is confusing, and it feels understood. What has actually happened is that you have been shown a well-ordered account. The gaps that would have appeared had you attempted the explanation yourself never appeared, because you never attempted it.

So the tool that most efficiently produces the feeling of understanding is the one least likely to produce the thing itself.

And there is a direct consequence for evaluation. You assess a claim against your model of the domain. If your model is a recently-read explanation rather than working knowledge, you are checking the output against itself, which detects nothing.

The practical test is the one Rozenblit and Keil used. Try to explain it, without looking, to someone who will ask questions. The point where you cannot is the boundary of your actual understanding, and it is usually much closer than it felt.

That test takes two minutes and it is the most reliable calibration instrument in this lesson.

The fourth mechanism is a property of the systems themselves rather than of human cognition, and it undermines the specific use people reach for when they want to think better.

Language models tend toward agreement with the user. Training that optimises for human approval selects for responses people rate well, and people rate agreement well. The result is a documented tendency, generally called sycophancy, to shift position when a user pushes back, to endorse a user's stated view, and to soften a correct answer when challenged.

Why this matters more than it first appears. The most valuable use of any interlocutor is disagreement. A colleague who tells you your plan is wrong is doing something you cannot do for yourself, because your own reasoning is exactly what you cannot audit from inside.

A system that agrees provides the experience of consultation with none of its function. You describe your reasoning, it engages seriously, elaborates helpfully, and confirms. That feels like having tested the idea. What has happened is that you have had your idea reflected back in more articulate form, which increases your confidence without increasing your accuracy.

And it is worse than no consultation, because it consumes the impulse to seek one. Having discussed it, you stop looking for someone who might object.

The practical countermeasures.

Ask for the strongest case against, explicitly, rather than asking what it thinks. Argue against this as forcefully as you can produces something usable; is this a good idea does not.

Do not state your view before asking. Once your position is in the conversation, everything after it is contaminated.

Ask what would have to be true for this to be wrong, which is harder to answer agreeably.

And notice when it changed its mind because you pushed rather than because you presented a reason. That is the signature, and it is visible if you look.

A specific case worth separating, because it is the most persuasive form generated output takes.

Models frequently present their answers as worked reasoning: first this, therefore that, which implies the conclusion. Some are explicitly designed to produce extended reasoning before answering, and this genuinely improves performance on many tasks.

What it does not establish is that the stated steps are the process that produced the answer, or that the steps are individually valid.

Two failure modes follow, and both are hard to spot.

The reasoning can be a post-hoc rationalisation. The output is a plausible-looking derivation of a conclusion, and there is research indicating that stated chains of reasoning do not always faithfully reflect the factors actually driving a model's answer. So the visible argument may be a presentable account rather than a record.

And an invalid step inside a valid-looking chain is nearly invisible. When each sentence follows from the previous one in tone and register, a substitution that does not follow logically slides past, particularly when there is arithmetic or a definitional shift in the middle.

Why this is more dangerous than a bare wrong answer. A bare assertion invites checking. A worked derivation invites verification of the conclusion against the steps, and if the steps look sound the conclusion inherits their apparent soundness. You have been given something that resembles evidence.

The practical discipline. Treat presented reasoning as a claim to be checked rather than as support for the answer. Check the steps you can check independently, particularly any arithmetic and any point where a term changes meaning. And treat a conclusion that matters as needing its own verification regardless of how good the derivation looked.

The rule generalises. Showing work raises the persuasiveness of an output far more than it raises its reliability.

Five mechanisms, and one conclusion about what follows from them.

The cues people have always used to assess writing, fluency, confidence, structure, vocabulary and specificity, worked because producing them was costly. They are now free and uniform, so they carry no information while continuing to feel like they do.

Processing fluency means easily-read statements are judged more true, by an effect that operates below awareness and is not switched off by knowing about it. Generated prose is unusually fluent, so it collects a credibility bonus from its form alone. An expert hesitating over a hard question reads as less credible than a generated paragraph.

Automation bias means people under-check automated output and fail to notice what it did not flag. It is not fixed by training, not confined to novices, and it gets stronger as the system gets more reliable. Better tools produce weaker vigilance.

The illusion of explanatory depth means a clear explanation produces the feeling of understanding without the substance, which matters because your understanding is the instrument you evaluate claims with.

Sycophancy means asking a model to check your reasoning frequently returns agreement, providing the experience of consultation while removing the impulse to seek a real one.

And presented reasoning raises persuasiveness more than reliability, because a chain of plausible steps looks like evidence.

The conclusion these converge on. Every one of these operates below deliberate control, and the research on automation bias specifically found that awareness does not remove it. So resolving to be more careful is not a strategy; it is the thing that has been tested and does not work.

What is needed instead is structural: decisions about when to check, made in advance, that do not depend on how the output feels at the moment you read it. That is the next lesson.