LLM Pretraining: Data, Loss, and What Actually Happens

Every major language model — GPT, Llama, Mistral, Gemini — is pretrained on a single objective: predict the next token. That's it. Given a sequence of tokens $x_1, x_2, \ldots, x_{t-1}$, the model outputs a probability distribution over the vocabulary for $x_t$. Training minimizes how wrong those predictions are across billions of such positions.

This deceptively simple objective forces the model to internalize grammar, facts, reasoning patterns, and world knowledge — because all of that information is latent in which token comes next. You cannot predict "The capital of France is ___" without knowing geography. You cannot predict the closing brace of a function without understanding syntax. The objective is humble; its consequences are not.

Formally, given a corpus of $N$ tokens, the training loss is: $\mathcal{L} = -\frac{1}{N} \sum_{t=1}^{N} \log P(x_t \mid x_1, \ldots, x_{t-1})$ This is the average negative log-likelihood over all token positions — also called cross-entropy loss.

Cross-entropy measures the surprise of seeing the true token under the model's predicted distribution. If the model assigns probability 1.0 to the correct token, $-\log(1.0) = 0$ — no surprise. If it assigns 0.01, $-\log(0.01) \approx 4.6$ nats — very surprised.

In practice, the model outputs a logit vector of size $|V|$ (vocabulary size, typically 32k–128k), which is passed through a softmax: $P(x_t = k) = \frac{e^{z_k}}{\sum_j e^{z_j}}$ The loss for that position is $-\log P(x_t = \text{true token})$.

A useful reference point: a random model over a 50k-token vocabulary gives $\log(50000) \approx 10.8$ nats. A well-trained model reaches ~2.0–2.5 nats on held-out English text. Perplexity is just $e^{\mathcal{L}}$ — a 2.0 nat loss means perplexity ≈ 7.4, meaning the model is roughly as uncertain as choosing uniformly among 7 tokens.

Heads up: Loss comparisons across models are only valid when tokenizers are identical. A model with a larger vocabulary will mechanically show lower per-token loss, even with the same underlying capability.

Common Crawl is a non-profit that has been crawling the web since 2008. Each monthly snapshot (a "crawl") contains ~3–4 billion pages in WARC format — compressed raw HTTP responses including HTML, headers, and metadata. The total archive exceeds a petabyte.

For LLM pretraining, you typically start from the WET files (plain text extracted from HTML) or re-extract text yourself using tools like trafilatura or resiliparse. Common Crawl is the upstream source for nearly every major open-weights pretraining dataset: The Pile, RedPajama, Dolma, RefinedWeb, FineWeb, and more.

The catch: raw Common Crawl is ~50% garbage. Boilerplate navigation menus, ad-injected pages, machine-translated spam, duplicate content, and toxic material are all in there. Naively training on raw WET files produces a noticeably worse model than training on the same volume of curated text. The curation pipeline is where the real engineering lives.

Duplicate documents inflate the apparent dataset size, cause the model to memorize specific sequences, and waste compute on redundant gradient signal. Two main strategies:

• Exact dedup: hash each document (MD5 / SHA-256), drop duplicates. Cheap and catches copy-pastes.
• Fuzzy dedup: MinHash + Locality-Sensitive Hashing (LSH). Builds a signature for each document from multiple hash functions, then buckets near-duplicates together. Catches paraphrased or slightly-edited copies. The standard tooling is datasketch or the C++ pipeline from the BigScience workshop.

The Pile team found ~30% of CommonCrawl is near-duplicate. FineWeb deduplication at the URL level + MinHash removed roughly half the data before quality filtering even ran. Lee et al. (2022, "Deduplicating Training Data Makes Language Models Better") showed that dedup improves downstream performance and significantly reduces verbatim memorization.

Gotcha: Dedup across train/test splits is critical. If your evaluation benchmark text appears verbatim in training data, benchmark scores are meaningless.

After dedup, a typical pipeline applies several filter passes:

Heuristic filters (fast, rule-based):

• Minimum token count (drop pages < 100 words)
• Maximum repetition ratio (flag pages where the same line repeats > 30% of content)
• Symbol-to-word ratio (drop pages heavy on #@!% — SEO spam)
• Language detection via fastText (keep only target languages)
• perplexity filter using a small KenLM model trained on Wikipedia — drop pages with very high perplexity (gibberish) or suspiciously low perplexity (boilerplate templates)

Classifier filters (slower, higher-signal):

• A fastText or small BERT-family model trained to classify "high quality" (Wikipedia, books) vs. "low quality" (random crawl). C4 used this; FineWeb uses a custom educational-quality classifier trained on GPT-4 annotations.

The FineWeb-Edu variant (HuggingFace, 2024) filters for educational content specifically and achieves better benchmark scores on knowledge-heavy evals even with fewer tokens — a strong argument that quality beats quantity past a certain scale.

The Pile (EleutherAI, 2020) was the first large-scale open pretraining dataset assembled with transparency. It combined 22 diverse data sources — Common Crawl (Pile-CC), books (Books3, Gutenberg), code (GitHub), academic papers (PubMed, ArXiv), legal documents (FreeLaw), and more — totaling 825 GB. Its multi-source design was intentional: diversity improves downstream generalization. GPT-Neo and GPT-J were trained on it.

RefinedWeb (Mistral/Falcon team, 2023) took a different bet: scale Common Crawl aggressively but filter harder, rather than mixing many sources. Using the MacroData Refinement (MDR) pipeline with strict dedup and heuristics, they extracted ~600B tokens that matched or beat multi-source blends on most benchmarks. Falcon-40B was trained on it.

FineWeb (HuggingFace, 2024) is the current open-source standard. It starts from 95 Common Crawl dumps (~15T tokens before filtering), applies URL dedup, MinHash, heuristic filters, and a custom quality classifier. The 10T-token release and 1.3T FineWeb-Edu subset are both publicly available on the Hub and reproduce Llama-3 level performance when used as the base corpus.

A pretraining loss curve plots cross-entropy (nats or bits-per-byte) on the y-axis against training steps or tokens processed on the x-axis. What you actually see:

• Sharp drop in the first ~1% of training: the model learns basic token co-occurrence statistics and common n-grams almost immediately.
• Long smooth power-law decay: consistent with the Chinchilla / Hoffmann et al. (2022) scaling laws — loss decreases roughly as $\mathcal{L} \propto N^{-0.076}$ in parameters and $D^{-0.095}$ in data.
• Bumps and plateaus: often correspond to learning rate schedule changes, data mixture shifts, or the model "discovering" a new capability (arithmetic, code patterns). Some bumps are artifacts of per-domain difficulty differences as the data pipeline cycles through shards.
• Train vs. eval divergence: a widening gap between train loss and held-out eval loss signals memorization — especially likely if dedup was skipped or the eval set overlaps with training data.

Loss alone doesn't predict downstream task performance. A model at 2.1 nats that trained on diverse domains often beats a 1.9-nat model trained on a narrow corpus.

Seeing the code makes the math concrete. Here is a stripped-down training step:

import torch
import torch.nn.functional as F

def training_step(model, input_ids, optimizer):
    # input_ids: [batch, seq_len]
    # targets are the same sequence shifted by one position
    inputs = input_ids[:, :-1]   # [batch, seq_len - 1]
    targets = input_ids[:, 1:]   # [batch, seq_len - 1]

    logits = model(inputs)       # [batch, seq_len-1, vocab_size]

    # Flatten for F.cross_entropy
    loss = F.cross_entropy(
        logits.reshape(-1, logits.size(-1)),  # [batch*(seq_len-1), vocab_size]
        targets.reshape(-1),                  # [batch*(seq_len-1)]
    )

    optimizer.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)  # gradient clipping
    optimizer.step()
    return loss.item()

Notice: the target at position $t$ is simply the input at position $t+1$. There is no separate label generation step — the supervision is baked into the sequence itself. F.cross_entropy applies softmax internally, so the model outputs raw logits.

Modern pretraining doesn't just dump all filtered text into a single stream. Data is organized by domain — web text, code, books, scientific papers, math — and each domain is sampled at a specified weight during training.

Weighting matters because domains differ in size and quality. If you sample proportionally to raw token count, code (relatively small) gets drowned out by web text. Upsampling code improves coding benchmarks substantially; upsampling academic text improves reasoning and STEM.

Recent work formulates this as an optimization problem. DoReMi (Xie et al., 2023, Stanford) trains a small proxy model and a reference model simultaneously to find domain weights that minimize worst-case perplexity across domains — then uses those weights for the large model. RegMix (2024) and similar methods use regression on small-scale experiments to predict which weights maximize downstream benchmark scores.

The practical takeaway: for a 1T-token run, getting domain weights right is worth as much as adding ~10–20% more tokens from the wrong distribution.

A few traps to avoid when evaluating claims about pretraining:

• Token count vs. parameter count confusion: Chinchilla showed you should train ~20 tokens per parameter for compute-optimal runs. A "7B model trained on 1T tokens" is 3× over-trained for compute but often optimal for inference cost.
• Perplexity on what?: WikiText-103 perplexity is almost meaningless for a 70B model. Always check the eval set.
• Data contamination: If a model reports strong MMLU or HumanEval scores, check whether those benchmarks appear in the training data. Many do. HuggingFace's lm-eval-harness now includes contamination detection.
• "We used open data" ≠ reproducible: The exact version of Common Crawl dumps, filter thresholds, and mixing weights are rarely published in full. FineWeb is a notable exception — the full pipeline code is public.
• Loss reported in different units: nats ($\log_e$) vs. bits ($\log_2$) differ by a factor of $\log_2(e) \approx 1.44$. Bits-per-character and bits-per-byte add another conversion. Always check the units before comparing numbers across papers.