Filtering: The Half That Decides Quality

The central discipline of synthetic data is that generation is cheap and inclusion is expensive.

Every example you keep becomes a training target. A wrong example does not merely fail to help; it actively teaches the student to reproduce the error, and it does so with the same weight as a correct one. Gradient descent has no way to know which is which.

So the correct posture is aggressive rejection. Generate four, eight, sixteen candidates per prompt and keep only those that survive. Because generation cost is roughly linear in samples while quality gain can be steep, the arithmetic usually favours generating far more than you need.

Recall the number from the first lesson. A generator correct 35 percent of the time yields at least one verified-correct answer on 96.8 percent of prompts at eight samples. The dataset you keep is dramatically better than the generator that produced it, and it cost eight times the tokens, which were nearly free.

The question is what plays the role of the checker, and the answers form a strict hierarchy of strength.

Where an answer can be checked by something that is not a language model, use that and stop looking for alternatives.

Code with tests. Run the generated function against the test suite. Passing is not an opinion.

Mathematics with a checkable result. Compare against a symbolic solver, or against the ground truth if the problem was constructed programmatically.

SQL. Execute both the generated query and a reference against the same database and compare result sets.

Structured extraction. Check every extracted value appears verbatim in the source document. This one is underused and catches fabricated field values immediately.

Formal statements. A proof assistant accepts or rejects.

What makes these categorically different is independence. The checker's judgement does not come from the model, so it supplies information the generation process did not contain. That is the mechanism from the first lesson that lifts the ceiling above the generator.

The strategic consequence is worth acting on: it is often worth reshaping a task so that verification becomes possible, because a verifiable formulation of a slightly narrower problem beats an unverifiable formulation of the exact one.

When no external checker exists but the task has a determinate answer, agreement across independent samples is a usable proxy.

Generate the same problem several times at non-zero temperature and compare the final answers. Where the samples converge, keep the answer. Where they scatter, discard the item.

The reasoning is that a model arriving at the same answer by several different routes is more likely to have derived it than a model producing one answer once. Incorrect answers tend to be incorrect in varied ways, so they fail to concentrate.

Two limits keep this honest. It only applies where answers can be compared for equality, which means extraction, classification and numeric results rather than open-ended prose. And it detects confidence rather than correctness: a model consistently wrong about something will agree with itself every time, and that is exactly the case where a systematic error enters the dataset with high confidence.

So self-consistency is a real filter and a weaker one, and it should never be described as verification.

For open-ended output, the practical filter is another model scoring the candidates. It works, and it has documented biases that have to be controlled rather than hoped away.

Position bias. When shown two candidates, judges favour one position over the other regardless of content. The control is to score each candidate in both orders and average, or to score candidates independently rather than in pairs.

Length bias. Judges systematically prefer longer answers, which is a problem when your filter is selecting training data, because the student then learns to be verbose. The control is to check whether score correlates with length across your accepted set, and to normalise if it does.

Self-preference. A judge tends to prefer text from its own model family, which matters when the same model generates and judges. Using a different family for judging is the cheap fix.

The non-negotiable step is calibration. Label a few hundred examples by hand, run the judge on them, and measure agreement. A judge that agrees with you 70 percent of the time is admitting 30 percent errors into training data, and you cannot know that without measuring.

Filters should run cheapest-first, because every candidate an early stage removes is a candidate an expensive stage does not have to score.

Heuristics come first and cost nothing: length bounds, repetition detection, refusal patterns, format validity, presence of the required fields. These remove a surprising share of candidates, and removing them before a judge call is pure saving.

Deduplication comes next, and near-duplicate detection matters far more here than on web data, because a generator produces genuinely similar outputs rather than copies. Exact matching catches almost none of it.

Verification comes third where it exists, and it is placed after dedup because running a test suite is more expensive than a hash comparison.

Judging comes last, applied only to whatever survives, since it is the most expensive stage and the least reliable.

Contamination checking sits outside the cascade, applied to the final set against your evaluation data, because it is the one check whose failure invalidates the entire result rather than degrading it.

Cheap filters are unglamorous and remove more bad data per unit effort than anything else in the pipeline.

Refusal detection. Generators decline, hedge, or produce a meta-comment about the request. Training on those teaches a model to refuse. A pattern list catches most of them.

Repetition. Degenerate loops where a phrase repeats until the token limit. Detect by measuring the ratio of unique n-grams to total n-grams and thresholding.

Length bounds at both ends. Very short responses are usually truncated or evasive. Very long ones are usually rambling or looping.

Truncation. A response ending mid-sentence hit the token limit, and training on it teaches the model to stop abruptly.

Format validity. If the output should be JSON, parse it. If it should cite a source, check the citation exists in the input.

Leakage of the scaffolding. Generated text sometimes includes the prompt, the instruction, or phrases like as an AI language model, or Sure, here is. Those teach the student to emit scaffolding, and they are trivially detectable.

Web-scale pretraining pipelines deduplicate because the internet contains literal copies. Synthetic pipelines deduplicate for a different reason, and the standard tooling does the wrong thing.

A generator rarely produces byte-identical outputs. It produces the same example with different names, numbers and phrasings, which exact matching and standard hashing both pass straight through.

The result is a dataset that reports zero duplicates and is, semantically, a few hundred examples repeated. Training on it overweights those patterns exactly as if they were literal copies.

So the tooling has to work on meaning. Embed every example and remove those whose nearest neighbour is closer than a threshold. Or use near-duplicate detection over shingles, which catches heavy paraphrase more cheaply than embeddings at large scale.

The threshold is a judgement call and worth calibrating by hand: too aggressive and you delete legitimate variations on a common task, too loose and near-copies survive. Inspect the pairs at the boundary rather than picking a number from a paper.

Run this before any expensive filter, since deduplicating first means judging fewer candidates.

Every other problem in this lesson makes a model worse. This one makes your measurements meaningless, which is more dangerous because it looks like success.

The mechanism is direct. A generator trained on public data has seen the public benchmarks. Asked to produce examples in a domain, it can reproduce benchmark items, sometimes near-verbatim. Those enter your training set. You then evaluate on that benchmark and the scores are excellent, because the model was trained on the test.

The signature is a model that performs well on public benchmarks and disappoints in production, which is a pattern common enough that it should be the first hypothesis rather than a surprise.

The check is mechanical and cheap. For every evaluation set you use, compute overlap against your training data: exact match on normalised text, near-duplicate detection, and embedding similarity above a threshold. Anything that hits gets removed from training, never from the evaluation.

The stronger discipline is a held-out set the generation process never touched, ideally collected after the generator's training cutoff, so contamination is impossible rather than merely checked for.

Filtering trades volume for quality, and the trade has a shape worth knowing.

Early rejection is nearly free. The first filters remove candidates that are clearly bad, and the dataset improves with no meaningful loss of coverage.

Middle rejection is the productive zone. Verification and calibrated judging remove genuinely wrong examples, and quality improves faster than volume falls.

Late rejection starts to hurt. Filters tuned to keep only the very best select for a narrow notion of good, and the surviving set becomes homogeneous. You have optimised for the judge rather than for the task, and the diversity you built in the previous lesson is quietly discarded.

That last failure is the one to watch, because it looks like success on every metric you are tracking: acceptance rate falls, average judged quality rises, and the dataset gets worse.

The control is to measure coverage after filtering, not before. Compare embedding spread of the accepted set against the generated set. If filtering collapsed the spread, it removed variety rather than errors, and the threshold is too tight.

A synthetic dataset without provenance is unauditable, and the metadata costs nothing at generation time and cannot be reconstructed later.

The generator: which model, which version, which date, under which terms. This is the licensing record from the first lesson and the input to any question about where a dataset came from.

The prompt and the seed that produced it, so a suspicious cluster of examples can be traced back to the template that generated them.

Which filters the example passed, and its judge score. When a trained model behaves oddly, being able to ask what the accepted set looked like at a given score threshold is the difference between diagnosis and guessing.

How many candidates were generated for this prompt and how many survived. A prompt with a very low survival rate is a signal about the task, not about the sample.

And the generation round, if the pipeline iterates, because that field is what makes the next lesson's distinction between accumulating and replacing possible to enforce.