Outliers: Why Large Models Resist Naive Quantization

The naive method is round-to-nearest: compute a scale from the observed range, divide, round, done. No calibration data, no optimisation, a few seconds of work.

Where it works and where it breaks is a clean pattern, and the pattern is the subject of this lesson.

That second failure is the strange one. Activations at 8 bits should be easy: 256 levels is generous for values that mostly sit within a small range. The reason it breaks is not about rounding at all. It is about what large transformers do to their own activation distributions.

Dettmers and colleagues documented the effect while developing LLM.int8(), and the shape of it is consistent enough to plan around.

Beyond roughly 6.7 billion parameters, transformers develop systematic outlier features in their activations. Three properties make them pathological for quantization:

• Large. Reported magnitudes reach around a hundred times the typical activation value, with other work observing per-channel ranges twenty to a hundred times the majority.
• Concentrated. The outliers do not scatter randomly. They occupy a small number of fixed dimensions of the hidden state, the same dimensions across inputs.
• Persistent. Those channels carry outlier values on almost every token, rather than on rare inputs you could treat as exceptions.

The useful summary is that variance across channels is enormous while variance across tokens within one channel is small. That asymmetry is not a defect to be trained away. It is the structure every serious activation quantization method is built around.

The damage is easy to see arithmetically, and doing so makes every later fix obvious.

Take a tensor whose values mostly sit between minus 1.5 and 1.5, and quantize symmetrically to 8 bits with 127 positive levels. The step size is 1.5 over 127, which is 0.0118. A value of 1.0 maps to level 85. Plenty of resolution.

Now introduce a single outlier of 100. The scale is set by the largest absolute value, so the step becomes 100 over 127, which is 0.787.

One number consumed the entire dynamic range, and it did so through the scale, not through the rounding.

The same reasoning explains an asymmetry that governs the whole field: weights quantize far more readily than activations.

Weights are static. They are fixed after training, so you can inspect the actual distribution, choose scales offline, and spend as much computation as you like getting it right. Nothing about the input changes them.

Weights are also better behaved. Their per-channel distributions are roughly bell-shaped around zero without the extreme fixed-channel structure activations show, so per-channel or per-group scaling already handles most of the range problem.

Activations are neither. They depend on the input, so scales must either be computed at runtime, which costs a reduction over the tensor before every matrix multiply, or estimated from calibration data, which risks a mismatch when real traffic differs.

This is why W4A16 is routine and W4A4 is a research frontier, and why every technique in the rest of this lesson exists to make activations behave more like weights.

The first response is the most direct. If a few channels are the problem, take them out of the quantized path.

LLM.int8() applies mixed-precision decomposition. It identifies the outlier dimensions, computes their contribution to the matrix multiply in 16-bit, computes everything else in 8-bit, and sums the two results. Because outliers occupy a small fraction of dimensions, almost all of the arithmetic still runs at 8 bits, and accuracy is preserved at essentially the 16-bit level.

The method is correct and its costs are structural rather than incidental. Splitting a matrix multiply into two paths with irregular, data-dependent column selection is unfriendly to hardware built for dense regular work. Kernels grow complex, memory access becomes scattered, and the realised speedup falls short of what the bit width suggests.

Decomposition is best understood as the correctness-first answer: it proved outliers were the cause, and it made 8-bit inference usable, while leaving the performance question to the approaches that followed.

The three families differ in what they do to the outlier, and understanding that distinction is enough to choose between them.

Decomposition leaves the outlier where it is and routes around it. The tensor is split, the awkward channels are computed at high precision, and the results are added. The distribution is unchanged; the pipeline works around it.

Smoothing shrinks the outlier by dividing its channel down, and compensates by scaling the matching weight row up. The product is identical, but difficulty has moved from the activation side to the weight side, where per-channel scaling already handles it.

Rotation removes the outlier's privileged position entirely. Multiplying by an orthogonal matrix mixes every channel into every other, so the large value is spread across the whole hidden dimension instead of concentrating in one place.

Only the last one actually changes the distribution being quantized. That is why rotation is the family that opened 4-bit activations, where the other two still struggle.

SmoothQuant, from Xiao and colleagues at ICML 2023, starts from an algebraic observation. In the product of an activation matrix and a weight matrix, you can divide activation channel j by any positive number and multiply the corresponding row of the weight matrix by the same number, and the result is unchanged.

So choose those numbers to make both sides easy. Divide the outlier channels down until their range resembles the others, and push the compensating factor into the weights, which tolerate a wider range because they are static and quantized per channel.

The scale for channel j is

Alpha controls the migration: zero leaves activations untouched, one moves all the difficulty to the weights, and 0.5 is the usual starting point.

The scales fold into the preceding layer's parameters offline, so at inference time the transformation is free. That is what makes W8A8 practical on hardware built for dense arithmetic.

Smoothing reduces the spike. Rotation removes the concept of a privileged channel altogether.

The idea, developed in work such as QuaRot, is to insert an orthogonal matrix and its inverse into the computation. Because the two cancel, the function the network computes is unchanged. But the values being quantized are now expressed in a rotated basis.

A rotation mixes every input channel into every output channel. A value a hundred times larger than its neighbours no longer sits alone in dimension 2047; its energy is distributed across the whole hidden dimension. The resulting distribution is closer to Gaussian and has a far smaller ratio between its largest and typical magnitudes, which is exactly what a uniform grid wants.

Randomised Hadamard transforms are the usual choice, because they can be applied in time proportional to n log n rather than n squared, and many rotations can be fused into adjacent weight matrices offline.

This is the family that made 4-bit activations tractable rather than merely conceivable.

One more piece of structure matters, and it is the reason serious formats are never a single bit width applied everywhere.

Different parts of a transformer tolerate quantization very differently. Attention projections and the layers at the very start and end of the network tend to be more sensitive than the wide feed-forward blocks in the middle, which contain most of the parameters and much of the redundancy.

That combination is fortunate. The sensitive tensors are a small share of total weight bytes, so raising their precision costs little, while the tolerant tensors are where the memory actually is.

The GGUF k-quant family in llama.cpp is built on exactly this. A file labelled Q4_K_M is not uniformly 4-bit. The K denotes the k-quant family, the M denotes the medium variant, and the scheme assigns higher precision to attention tensors while keeping feed-forward weights lower. The average lands near the nominal bit width while the damage concentrates where it matters least.

When you see a mixed-precision recipe, this is the observation it encodes.

The families are not competitors so much as answers to different questions, and modern pipelines combine them.

A practical ordering. If you only need weight-only quantization, none of this is required: go straight to the algorithms in the next lesson. If you want W8A8 for compute-bound serving, smoothing is the standard route and is usually built into the toolchain. If you are pushing to 4-bit activations, rotation is what makes it work.

And across all three, the sensitivity structure argues for mixed precision rather than one bit width everywhere.