What Quantization Actually Does to a Number

Generating one token from a transformer requires reading every weight from memory once. Not once per batch item, once per forward pass. At batch size one the arithmetic units are idle most of the time, waiting on memory.

That makes decoding memory-bandwidth-bound, and it gives quantization an unusual property. Compression normally costs speed: you trade cycles to save space. Here it buys both. Fewer bytes per weight means fewer bytes to move, which means more tokens per second.

Put numbers on it. A 70-billion-parameter model at 16 bits per weight is 140 GB. On a GPU with roughly 3 TB/s of memory bandwidth, the ceiling is about 21 tokens per second, before any compute at all.

The same model at 4 bits is around 36 GB, and the ceiling rises to about 82 tokens per second. Same hardware, same model, roughly four times the throughput ceiling, and it now fits where it previously did not.

Quantization maps a continuous range onto a small set of integers. The standard form is affine.

Here s is the scale, a positive floating point number giving the spacing between adjacent levels, and z is the zero point, an integer offset that lets the grid represent an asymmetric range.

Two variants matter. Symmetric quantization sets z to zero and derives the scale from the largest absolute value: s equals absmax divided by the largest positive level. The grid is centred on zero, which is a good fit for neural network weights, and dequantization is a single multiply.

Asymmetric quantization derives the scale from the full range, s equals max minus min divided by the number of levels minus one, and picks z so that real zero maps exactly onto a grid point. That costs an extra subtraction but wastes no levels when the distribution is one-sided, as after a ReLU.

Take eight weights and quantize them symmetrically to 4 bits, using levels minus 7 through 7.

The values: 0.12, -0.35, 0.87, -1.42, 0.05, 0.63, -0.21, 1.10. The largest absolute value is 1.42, so the scale is 1.42 divided by 7, which is 0.202857.

Divide each weight by the scale and round.

Root mean squared error: 0.0545.

Two things to notice. The value that set the scale is reproduced exactly, and everything else absorbs the error. And 0.05 collapsed to zero, because it fell below half a step. Small weights near a large one do not survive.

The previous example used one scale for eight weights. Real implementations choose a granularity, and it is the single most consequential knob after bit width.

Per-tensor uses one scale for an entire weight matrix. Cheapest in metadata, worst in accuracy, because one extreme value sets the step size for millions of weights.

Per-channel uses one scale per output channel, typically a row or column. Standard for weights, since each output neuron gets its own range.

Per-group subdivides further, one scale per contiguous block of 32, 64 or 128 weights. This is what makes 4-bit weight quantization viable, because an unusually large weight now only damages its own group.

Metadata is not free. With a 16-bit scale and a 4-bit zero point per group, the true cost per weight is 4 plus 16 over the group size plus 4 over the group size. At group size 128 that is 4.156 bits. At 64 it is 4.313.

The notation WxAy names the bit width of weights and of activations. It tells you what a scheme actually buys.

W4A16 is weight-only quantization: weights are dequantized back to 16 bits inside the kernel, immediately before the matrix multiply, so the arithmetic is unchanged. This is what GPTQ, AWQ and most GGUF files give you. W8A8 lets the multiply itself run on integer or 8-bit-float tensor cores, which matters for prefill and for large batches, which are compute-bound rather than bandwidth-bound. W4A4 requires the outlier handling the next lesson covers.

The practical consequence: if your workload is single-stream chat, weight-only is enough. If you are serving high-concurrency traffic or processing long prompts, activation quantization is where the remaining gain lives.

The difference between weight-only and full quantization is a question of where in the pipeline values return to high precision.

In a W4A16 path, the packed 4-bit weights are read from memory, unpacked and multiplied by their group scales inside the kernel, and the matrix multiply runs in 16-bit. The bandwidth saving is real because only 4-bit data crossed the memory bus. The compute saving is zero because the multiply is unchanged.

In a W8A8 path, activations are quantized on the fly, the multiply runs natively on 8-bit tensor cores, and the integer accumulator is rescaled once at the end. Both bus and arithmetic units benefit.

The diagram makes the asymmetry visible. Weight-only narrows one arrow. Full quantization narrows the arrow and swaps the engine.

This is also why a 4-bit model can be slower than an 8-bit one at high batch sizes: the dequantization work is per-weight, and once you are compute-bound rather than bandwidth-bound, it is pure overhead.

Uniform integer levels are the obvious choice, and often the wrong one, because weight distributions are not uniform. They are roughly bell-shaped around zero, so uniform spacing wastes levels in the sparse tails and starves the dense centre.

NF4, introduced with QLoRA by Dettmers and colleagues, addresses this directly. It is a 4-bit NormalFloat type built by quantile quantization: the 16 levels are placed so that a normally distributed input lands in each bin equally often. For weights that really are near-Gaussian, this is information-theoretically optimal, and it beats uniform INT4 at the same bit width for free.

Floating point grids are the other family. FP8 comes in two encodings: E4M3, with four exponent and three mantissa bits, representing roughly plus or minus 448; and E5M2, with more range, roughly plus or minus 57344, and less precision. The convention is E4M3 for weights and activations, E5M2 where the wider dynamic range of gradients is needed.

Floating point grids spend their resolution proportionally, which suits values spanning several orders of magnitude.

A format only pays off if silicon executes it. That constraint has shaped the recent generation of 4-bit formats.

Microscaling formats combine a low-precision element type with a shared scale over a small block, putting group-wise quantization directly into the number format rather than into software around it.

MXFP4 encodes blocks of 32 elements with an 8-bit exponent-only scale, of type E8M0. NVFP4 uses blocks of 16 with an E4M3 scale, trading more metadata for finer granularity and better accuracy.

NVIDIA's Blackwell generation supports FP4 natively in its fifth-generation tensor cores, alongside FP16 and FP8, with accumulation in higher precision. Where earlier 4-bit work was weight-only, unpacked into 16-bit before the multiply, hardware here executes the low-precision multiply directly.

The lesson generalises beyond any one vendor. Check what your target accelerates natively before selecting a format. A scheme with better paper accuracy that must be emulated in software will usually lose to a slightly worse one the hardware runs directly.

Put the arithmetic together for a 70-billion-parameter model, counting only weights.

At 16 bits: 70 billion times 2 bytes is 140 GB. That needs at least two 80 GB accelerators, so you pay for tensor parallelism and the interconnect it demands.

At 8 bits: 70 GB. One 80 GB accelerator holds the weights, leaving roughly 10 GB for KV cache and activations, which is tight but workable.

At 4 bits with group size 128: 70 billion times 4.156 bits over 8 is about 36.4 GB. Now more than half the device is free for KV cache, which is what actually determines how many concurrent requests you can serve.

That last point is the one people miss. Quantizing weights does not only make the model fit. It converts saved weight memory into KV cache capacity, and KV cache capacity is concurrency. The throughput gain from serving four times as many simultaneous sequences usually dwarfs the per-token bandwidth gain.

Three neighbouring techniques get confused with quantization, and keeping them separate keeps the decisions clean.

Quantization produces the same model at lower fidelity, with no training required in the post-training case. Pruning changes the shape of the computation; quantization changes its precision.

The practical significance is that these compose. You can distil, then quantize the student. You can prune, then quantize what remains. And their failure modes differ: a quantized model degrades smoothly and predictably in most cases, while an over-pruned one tends to fail abruptly.