Measuring the Damage, and Shipping It

Perplexity is the default quantization metric because it is cheap, needs no labels, and produces a single number. It is also close to useless for the decision you are making.

The problem is averaging. Perplexity measures how surprised the model is by each token in a corpus, then averages. The overwhelming majority of tokens in any text are easy: function words, continuations of common phrases, the second half of a name. A model that has lost real capability still predicts those perfectly, so they dominate the average and hide the loss.

Quantization damage concentrates in exactly the opposite place. It shows up on the decisions that were marginal to begin with, where the correct token was winning by a small margin and a little numerical noise flips it.

The practical consequence is a trap that catches people repeatedly. A quantized model can show a perplexity increase of a fraction of a percent and lose a large share of its accuracy on a task that matters. Reporting the perplexity delta is not evidence of safety.

The replacement is a task evaluation drawn from the work the model actually does, and a few design choices decide whether it can see anything.

Use your own traffic. Sample real prompts, in the real template, with the real system prompt and tool definitions. A public benchmark measures a different distribution than the one you serve.

Fix everything except the quantization. Same prompts, same seed, temperature zero. You are measuring one variable, and sampling noise is large enough to swallow the effect you are looking for.

Weight the hard cases. Since damage concentrates on marginal decisions, an evaluation set of representative traffic will understate it. Include a deliberately difficult slice: long contexts, ambiguous instructions, unusual formats, the tickets your support team escalates.

Evaluate several bit widths at once. A single comparison of 16-bit against 4-bit tells you a difference exists. A curve across 16, 8, 4 and, if relevant, lower tells you where the cliff is, which is the number you need to make a decision.

A COLM 2025 study of quantized reasoning models gives the most useful published map of the terrain. It evaluated the DeepSeek-R1-Distilled Qwen and LLaMA families from 1.5B to 70B, along with QwQ-32B and Qwen3-8B, across weight, KV cache and activation quantization, on mathematical, scientific and programming reasoning benchmarks including AIME, MATH-500, GPQA and LiveCodeBench.

The headline result is reassuring at the top and cautionary below it. Essentially lossless quantization is achievable at W8A8 or W4A16. Below those, accuracy risk becomes significant rather than negligible.

That is a useful default. Weight-only 4-bit and full 8-bit are the two settled operating points, and everything more aggressive requires you to measure rather than assume.

The study also identifies what predicts the outcome: model size, model origin and task difficulty. Smaller models have less redundancy to spare and degrade sooner, and harder tasks degrade before easy ones on the same model. Both are consistent with damage concentrating on marginal decisions.

The same study identifies reasoning models as being at higher risk, and the mechanism is worth understanding because it generalises to any long-horizon generation.

A quantized model makes a slightly noisier decision at each token. Over a short answer, that noise mostly cancels or goes unnoticed. Over a chain of thought thousands of tokens long, errors compound: one wrong intermediate step is conditioned on by everything after it, and the model will confidently continue down the branch it took.

KV cache quantization compounds along two axes at once. Error accumulates through model depth, layer to layer, and through sequence length, token to token. A perturbation that is negligible for a single cached token becomes substantial across thousands of them.

The operational reading is that your evaluation must exercise the generation length you actually serve. Testing a model that will produce long derivations, agent trajectories, or extended tool-use sequences on short question-and-answer pairs will systematically understate the damage.

Weight quantization and KV cache quantization are independent decisions with different arithmetic, and conflating them causes bad sizing.

The cache holds one key and one value vector per layer per token. Its size is two, times the number of layers, times the number of key-value heads, times the head dimension, times bytes per element.

For a 70B-class model with 80 layers, 8 key-value heads under grouped-query attention, and head dimension 128, that is 320 KB per token at 16 bits. An 8192-token sequence costs 2.68 GB. At 32768 tokens it is 10.74 GB, and at 131072 tokens 42.95 GB, for a single request.

Grouped-query attention already did the heavy lifting here. The same model with 64 key-value heads would need 2560 KB per token, eight times more.

On the accuracy side the evidence is encouraging down to a point and then abrupt. In the COLM study, moving the cache from 16 bits to 4 caused almost no degradation for Qwen3-8B and LLaMA3-8B. Moving it to 2 bits produced average drops of roughly 15.2 and 10.2 points.

Device memory splits three ways, and quantization acts on the first two independently.

Weights are fixed and known before you start. Quantizing them is a one-time decision that frees a fixed amount.

The KV cache is variable and is what limits concurrency. It grows with the number of active sequences multiplied by their lengths, and it is the only term that responds to traffic.

Workspace covers activations, attention scratch space, the runtime context and fragmentation. It is smaller than the other two but not negligible, and sizing plans that ignore it fail at the worst moment.

The diagram shows why weight quantization pays twice. The direct benefit is fewer bytes read per token, which raises the bandwidth ceiling. The indirect benefit is larger and usually dominant: every gigabyte freed from weights becomes a gigabyte of KV cache, which becomes more concurrent sequences, which becomes throughput.

Quantizing the cache attacks the same term from the other side, and the two multiply.

Put it together on a single 80 GB accelerator with the 70B-class model from earlier.

At 16-bit weights the question does not arise: 140 GB does not fit, so you are paying for two devices and an interconnect.

At 4-bit weights with group size 128, weights occupy about 36.4 GB. Reserve roughly 4 GB for workspace and runtime, and 39.6 GB remains for KV cache.

At 320 KB per token, that budget holds about 120,000 cached tokens: roughly 14 concurrent sequences at 8k context, or under 4 at 32k.

Quantize the cache to 8 bits and it holds about 241,000 tokens, so roughly 29 sequences at 8k. At 4 bits, about 483,000 tokens, or roughly 59 sequences at 8k and 14 at 32k.

That is the whole argument in one calculation. Weight quantization moved the model from not fitting to fitting. Cache quantization moved concurrency from 14 to 59 on identical hardware, a fourfold change in how many users one device serves.

Quantization is not uniformly a speedup, and the exception surprises people who benchmarked at batch size one.

Decoding is memory-bandwidth-bound. Every weight is read once per token regardless of batch size, so fewer bytes per weight translates almost directly into more tokens per second. This is where weight-only quantization shines.

Prefill is compute-bound. Processing a long prompt is a large matrix multiply with high arithmetic intensity, so the bottleneck is arithmetic throughput, not bytes. Weight-only quantization does nothing for it, and can hurt: unpacking 4-bit weights to 16 bits is per-weight work that buys nothing when you were not waiting on memory.

Large batches behave like prefill. As batch size grows, weights are amortised across more sequences and the regime shifts toward compute-bound, so the dequantization overhead becomes visible.

The result is a genuine crossover. A W4A16 model can be faster than W8A8 at batch size one and slower at batch size 64. Benchmark at the concurrency you will actually run.

An offline evaluation catches the large regressions. It will not catch everything, because your evaluation set is a sample of a distribution you only partly understand. The rollout is the second instrument.

Shadow first. Send a copy of live traffic to the quantized model without returning its answers, and compare outputs against the current model. This surfaces the input types your evaluation set did not contain, at no user risk.

Then canary on a small traffic share, with the ability to revert in one step. Keep both weight sets loadable for the duration.

Watch for the failure signatures quantization actually produces, which are not general incoherence. Increased repetition and looping. Malformed tool calls and broken JSON, since structured output depends on marginal token decisions. Degradation that grows with context length. And a rising rate of refusals or abandoned reasoning on hard inputs.

Each of those is a downstream symptom of noisier decisions at the margin, which is why they appear before anything that looks like a general quality drop.

Quantization is a trade, and a trade needs both sides quantified. A defensible report is short and contains six things.

The configuration, precisely enough to reproduce: model and revision, method, bit width, group size, what was left in higher precision, and the calibration set.

The accuracy curve across bit widths on your own evaluation, with the hard slice reported separately from the average, since that separation is where the decision usually turns.

The performance measurement at the concurrency you serve, not at batch size one: tokens per second, time to first token, and achieved throughput.

The memory budget, showing weights, cache and workspace, and the concurrency it implies.

The hardware assumption, naming what the target accelerates natively, because a format the silicon does not execute will not deliver its paper numbers.

And the recommendation with its threshold: what accuracy loss was accepted, on which slice, for what throughput gain.

That last line is what turns a benchmark into an engineering decision someone else can audit.