LLM Scaling Laws: From Kaplan to Chinchilla and Beyond

Before 2020, scaling up language models felt more like alchemy than engineering. You trained bigger models and hoped for the best. Kaplan et al.'s 2020 paper from OpenAI changed that by showing that loss follows power laws with respect to three quantities: model parameters $N$, dataset tokens $D$, and compute budget $C$.

This turned model development into something approximating a science. If you know your compute budget, the laws tell you roughly where to spend it — how large a model to train and on how many tokens. The 2022 Chinchilla paper from DeepMind then showed OpenAI had the ratio badly wrong, triggering a rethink across the entire field.

Understanding both papers is now essential context for anyone making decisions about training or deploying large models.

The core finding of Kaplan et al. ("Scaling Laws for Neural Language Models", OpenAI, 2020) is that test loss $L$ scales as a power law in each variable when the others are held fixed:

$L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N}, \quad L(D) \approx \left(\frac{D_c}{D}\right)^{\alpha_D}$

where $\alpha_N \approx 0.076$ and $\alpha_D \approx 0.095$ are empirical exponents, and $N_c$, $D_c$ are fitted constants. The exponents are small — doubling $N$ cuts loss by only ~5% — but they are remarkably stable across six orders of magnitude of scale.

The compute-loss relationship combines both: $L(C) \propto C^{-0.050}$. The key policy implication Kaplan drew was that for a fixed compute budget, you should favor larger models trained on fewer tokens — a conclusion Chinchilla would directly contradict.

Hoffmann et al. ("Training Compute-Optimal Large Language Models", DeepMind, 2022) ran a much broader sweep of model sizes and data quantities under matched compute budgets and found that Kaplan's advice was off. Their main result: for compute-optimal training, you should scale $N$ and $D$ equally.

The rule of thumb that emerged: roughly 20 tokens of training data per parameter. Concretely, a 70B-parameter model should train on ~1.4T tokens — the configuration they named Chinchilla. At the same compute budget as Gopher (280B parameters, 300B tokens), Chinchilla beat it decisively on nearly every benchmark.

Heads up: the "20 tokens per parameter" figure is a useful heuristic, not a law. The exact optimal ratio shifts with the exponents, which themselves depend on architecture and data quality.

The implication was brutal for the industry: GPT-3 (175B params, 300B tokens) was substantially undertrained.

Hoffmann et al. modeled loss as a function of both $N$ and $D$ simultaneously:

$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$

where $E$ is irreducible loss (entropy of natural language), and $A$, $B$, $\alpha \approx 0.34$, $\beta \approx 0.28$ are fitted constants. Compute is constrained by $C \approx 6ND$ (six FLOPs per token per parameter for a transformer forward + backward pass).

Minimizing $L$ subject to $C = 6ND$ gives the optimality condition:

$N_{\text{opt}} \propto C^{0.5}, \quad D_{\text{opt}} \propto C^{0.5}$

Both scale with $C^{0.5}$, hence equal scaling. Kaplan's exponents yielded $N_{\text{opt}} \propto C^{0.73}$, which over-weighted parameters. The discrepancy likely came from Kaplan holding learning rate schedules fixed — which artifically made more tokens look less useful.

If Chinchilla defines the compute-optimal point, why do labs train Llama 3.1 8B on 15T tokens — more than 500 tokens per parameter? Because compute-optimal for training loss is not the same as optimal for deployment.

The key insight: inference is not free. Once a model is deployed, it serves billions of requests. A smaller, longer-trained model has lower loss than a larger undertrained model at the same inference compute cost. Users pay per token at inference time, not per FLOP during training.

So the real objective function is:

$\text{minimize } L \text{ subject to fixed inference FLOP budget per query}$

not "fixed training FLOP budget". This flips the recommendation: overtrain a small model so it gets smarter per inference dollar. Llama models, Mistral, Phi — nearly every publicly-released model since 2023 is heavily overtrained relative to Chinchilla.

The train-longer trend has a parallel on the inference side: spending more compute at inference time rather than baking everything into weights. Techniques include:

• Chain-of-thought prompting — forces the model to produce intermediate reasoning tokens before answering
• Best-of-N sampling — generate $N$ completions, score with a reward model, return the best
• Tree search / MCTS — explore a tree of partial completions guided by a value function (used in AlphaCode 2)
• Speculative decoding — use a small draft model to propose tokens, verified in parallel by the large model

OpenAI's o1 and o3 models made inference-time scaling the headline story of 2024. The scaling law for inference compute is empirically log-linear in many reasoning benchmarks — you can keep spending and keep improving, up to a point.

Heads up: inference-time compute doesn't replace training compute. It leverages a well-trained model's latent knowledge; an undertrained model wastes the extra thinking budget.

Scaling laws are empirical fits — they extrapolate but can break. Several known limitations:

Data quality dominates at high D. Once you've swept the internet, more tokens means lower-quality text. The effective exponent $\beta$ degrades, so real curves bend away from the power law. Llama 3's 15T token dataset involved aggressive deduplication and quality filtering specifically to stay on the curve.

Architecture matters. The constants $A$, $B$, $\alpha$, $\beta$ are not universal. Mixture-of-Experts models have different scaling behavior because their effective parameter count differs from their active parameter count. Chinchilla's numbers were fit on dense transformers.

Emergent abilities don't follow smooth curves. Some capabilities (multi-step arithmetic, chain-of-thought reasoning) appear abruptly at certain scales rather than improving smoothly. Whether this is real emergence or a measurement artifact is still debated (Schaeffer et al., 2023 argue it's the latter).

The irreducible loss floor $E$ is non-trivial. Natural language has genuine entropy; no model can get loss below it.

You don't need to refit the curves yourself — but you do need to read them critically. Here's a minimal Python sketch for estimating compute-optimal $N$ given a FLOP budget $C$:

import numpy as np

# Chinchilla-fitted constants (Hoffmann et al. Table A3, IsoFLOP approach)
A, B = 406.4, 410.7
alpha, beta = 0.34, 0.28

def chinchilla_optimal(C_flops: float) -> tuple[float, float]:
    """Return (N_opt, D_opt) for a compute budget in FLOPs."""
    # From the closed-form solution of the constrained minimization
    N_opt = (A * alpha / (B * beta)) ** (1 / (alpha + beta)) * (C_flops / 6) ** (beta / (alpha + beta))
    D_opt = C_flops / (6 * N_opt)
    return N_opt, D_opt

C = 6e23  # ~GPT-3 compute
N, D = chinchilla_optimal(C)
print(f"Optimal params: {N/1e9:.1f}B, optimal tokens: {D/1e9:.0f}B")
# Optimal params: 67.5B, optimal tokens: 1490B

GPT-3's actual config was 175B params / 300B tokens — wildly off the Chinchilla optimum for that budget.

What does overtraining look like in practice? Compare these public models at roughly similar inference cost (7–8B active parameters):

The trend is unmistakable. Phi-3 Mini is the extreme case — Microsoft trained a tiny model on heavily curated "textbook quality" data, targeting users who need to run models on-device where inference cost is the binding constraint.

The lesson: Chinchilla optimality is a training-time concept. Real deployment decisions require you to plug in your actual inference-to-training cost ratio and solve accordingly.

Scaling laws describe loss on next-token prediction. Downstream task performance is messier — sometimes a 2x compute increase yields near-zero benchmark improvement, then another 2x causes a jump. Several open questions shape active research:

Data scaling walls. High-quality internet text may be genuinely finite. Synthetic data (generated by stronger models) is now a primary lever; its scaling behavior is still being characterized.

Multimodal scaling. Do the same power laws hold when you add image, audio, or code tokens? Early evidence (Flamingo, GPT-4V, Gemini) suggests yes, with separate constants per modality.

Scaling inference, not training. The o-series models suggest that for reasoning tasks, inference compute may be more elastic than training compute — you can keep improving by thinking longer, even with a frozen model.

Post-training amplification. RLHF and DPO can extract substantially more capability from a given pretrained checkpoint. Whether this shifts the optimal pretraining point is an open empirical question.

The meta-lesson: treat scaling laws as a compass, not a map.