Moderation and AI detection, and why both fail

Moderation starts from a constraint, not a preference. The volume uploaded to a large platform is far beyond what any number of humans could individually review. There is no budget, and no labour pool, that makes human-first moderation possible.

So the pipeline is always the same shape:

1. Classifiers score everything. Every item, automatically, in milliseconds.
2. The confident majority is auto-resolved. Clearly fine: published. Clearly violating: removed.
3. The uncertain middle escalates to humans. A tiny fraction of the total, which is the only reason human review is affordable at all.

Humans are not the first line. They are the appeals court for the ambiguous band, and the classifier decides who gets there.

This matters for a reason people miss: when a decision feels obviously wrong to you, it usually was not a bad judgement by a person. It was a threshold, applied automatically, to an item that never reached a human at all.

Two ways to be wrong, and they pull against each other.

• False positive: innocent content removed. Measured by precision, how much of what you removed deserved it.
• False negative: harmful content left up. Measured by recall, how much of the harm you actually caught.

You cannot maximise both with an imperfect classifier. There is one dial, the threshold, and it slides between them:

threshold LOW      -> catches more harm, removes more innocent posts
                      high recall, low precision

threshold HIGH     -> removes fewer innocent posts, misses more harm
                      high precision, low recall

The crucial consequence: where the dial sits is a value judgement, not a technical fact. No amount of model improvement chooses it for you. A platform setting it for self-harm content and for copyright will pick different points, because the cost of each error differs. Someone decides how many innocent creators is worth how much prevented harm, and that decision is unavoidable.

Classifiers judge an item. Policies are about meaning, and meaning does not live inside the item.

The same footage of violence is a war crime being documented, a news report, or glorification, depending entirely on who posted it and why. The same slur is an attack or a quotation in an anti-racism talk. The same medical image is educational or not depending on the surrounding context.

A model looking at pixels and audio cannot see intent, authorship, or the surrounding conversation. It sees the artifact. So it systematically mishandles exactly the cases where context does the work, which are also the cases people care most about.

This is why documentation of atrocities gets removed by systems trained to remove graphic violence: the classifier is correct that the content is graphic violence, and the classifier's correctness is not the question.

No threshold fixes this, because it is not a calibration error. The information required to make the decision was never in the input.

As generation got cheap, a new demand appeared: tell me whether a human made this. The tools sold for that job are statistical classifiers, and the published evidence on them is unusually damning.

The most direct source is OpenAI, reporting on its own product. Its AI Text Classifier identified just 26 percent of AI-written text, while flagging 9 percent of human writing as AI. OpenAI withdrew the tool in July 2023, citing its "low rate of accuracy". It was also unreliable below about 1,000 characters, which is most social posts.

Independent testing agrees. Weber-Wulff and colleagues tested 14 tools, including Turnitin, in the International Journal for Educational Integrity (2023), and concluded that "the available detection tools are neither accurate nor reliable". They found the tools lean toward labelling AI text as human, and that obfuscation techniques degrade them further.

Sit with that 9 percent. Applied to a class of 30, it is roughly three innocent people accused per assignment. Applied to a platform's uploads, it is millions.

The problem is not that the tools do nothing. It is that they are nowhere near accurate enough to justify a consequence.

An accuracy problem becomes a fairness problem once you see who absorbs the errors.

A Stanford study published in Patterns (Liang et al., 2023) ran 91 TOEFL essays written by non-native English speakers through seven commercial detectors. 61.3 percent were flagged as machine-written. Essays by native speakers were flagged around 5.1 percent of the time, and the same tools were near-perfect on US eighth-grade essays.

So the detectors were not uniformly bad. They were roughly fine for one group and badly wrong for another, which is worse, because it looks reliable right up until it destroys somebody.

The cause is mechanical, not political. Detectors lean on perplexity: roughly, how surprising each next word is. Writing that uses a narrower vocabulary and more predictable constructions scores low perplexity, and low perplexity is precisely the signature these tools read as machine-written. Non-native writing tends to be more regular, so it trips the wire.

The honest description: a detector does not separate machine text from human text. It separates statistically unsurprising text from surprising text. A person then reads that as guilt.

Detection asks an unanswerable question: does this artifact look synthetic? Provenance asks a tractable one: what does this file say about where it came from, and is that claim signed?

Two mechanisms, working differently.

C2PA Content Credentials attach cryptographically signed metadata recording how a file was made and edited. Backed by a coalition including Adobe, Microsoft, the BBC and camera makers such as Leica and Sony. A camera can sign an image at capture; each edit appends to the manifest.

Watermarking, such as Google's SynthID, embeds a statistical signal into the pixels or tokens themselves rather than into metadata. Because it lives in the content, it survives screenshots and re-encoding that destroy metadata.

The philosophical shift is the point. Detection guesses from the artifact and is wrong constantly. Provenance carries a claim from the source and can be verified. It only works if the claim was attached at creation, which is the whole difficulty.

Both approaches have holes, and they are structural rather than bugs to be patched.

Metadata is fragile. C2PA credentials live in metadata, and metadata is destroyed by screenshots, format conversion, and the re-encoding platforms routinely apply on upload. A genuine camera-signed photo can therefore arrive with its manifest stripped, so it verifies as unknown, which a careless reader treats as suspicious.

Coverage is the killer, and it is structural. A watermark exists only if the model that generated the content chose to embed one. Google has deployed SynthID across its own models, but there is no cross-vendor standard that every generator implements, and anyone running an open-weights model on their own hardware can simply decline to participate. Watermarking is therefore opt-in by the party with the least incentive to opt in.

The consequence follows without needing a statistic: a large share of generated content carries no provenance signal at all, and no detector can conjure one that was never embedded.

Absence proves nothing. No credential means unknown, not human-made. That is exactly the ambiguity a bad actor wants, and it costs them nothing: a screenshot produces it for free.

Put the pieces together and some firm conclusions fall out.

The honest position: origin is becoming unprovable by inspection, and the replacement is not a better detector but a chain of signed claims, which is only as good as its adoption.

For you as a poster, the practical takeaway is narrow and useful: disclose synthetic media yourself, keep provenance intact when a platform allows it, and never accept a detector's verdict, about you or about anyone else, as evidence of anything.

Detection inspects the artifact and guesses, producing false positives on genuine work. Provenance verifies a signed claim attached at creation, which is sound when present but frequently absent or stripped.