GPTQ, AWQ, and What Calibration Data Buys You

Round-to-nearest minimises the difference between each weight and its quantized version. That is the obvious objective and it is not the one you care about.

What you care about is that the layer produces the same output. Written down, the goal is to choose the quantized weight matrix that minimises the squared difference between the original layer output and the quantized layer output, over the inputs the layer will actually see.

Those are different problems, and the gap between them is where the whole field lives. A weight can be perturbed a great deal with little effect on the output, if the inputs it multiplies are usually near zero. Another can be perturbed slightly and matter enormously, if it multiplies a large, consistently present activation.

Round-to-nearest cannot see that distinction, because it never looks at the input. To see it you need examples of the input, which is what a calibration set is, and an algorithm that uses them. GPTQ and AWQ are two different answers to that.

GPTQ, from Frantar and colleagues, descends from Optimal Brain Quantization, itself a descendant of the Optimal Brain Surgeon pruning work. The core move is compensation.

Process the weight matrix column by column. Quantize one column, which introduces a known error. Then, before touching anything else, update all the columns not yet quantized so that they partially cancel that error at the output.

The update direction comes from the Hessian of the layer-wise objective, which for a squared output error is determined by the calibration activations: it is proportional to the input matrix times its own transpose. The inverse of that Hessian says how to distribute the error you just made across the weights you have not yet fixed.

By the end, no individual weight is optimally rounded, and that is the point. Each rounding decision was made knowing what the previous ones cost, so the accumulated output error is far smaller than independent rounding would give.

The word Hessian makes this sound more exotic than it is. Here it is a correlation matrix of the layer's inputs, and it answers a specific question: if I perturb weight i, how much does the output move, and which other weights could absorb that movement?

The diagonal is a sensitivity ranking. Input dimensions that are consistently large produce large diagonal entries, so weights multiplying them are treated as expensive to disturb.

The off-diagonal entries are what make compensation possible. They record that two input dimensions are correlated, which means an error introduced on one weight can be partly undone by adjusting the weight attached to the other.

Two implementation details exist for practical reasons rather than mathematical ones. The inverse Hessian is obtained through a Cholesky decomposition, because repeatedly updating an explicit inverse is numerically unstable at this scale. And columns are processed in blocks, with updates applied lazily to the rest of the matrix, because doing it strictly one column at a time is memory-bandwidth-bound and slow.

AWQ starts from a different observation. Not all weight channels matter equally, and the ones that matter are identifiable from a much simpler signal.

The signal is activation magnitude, not weight magnitude. A weight channel is salient if the activations it multiplies are consistently large, because errors there propagate. Roughly one percent of channels account for a disproportionate share of the damage.

The obvious response, keeping that one percent in 16-bit, works but produces a mixed-precision layout that hardware dislikes, for the same reason decomposition was awkward in the previous lesson.

AWQ takes the algebraic route instead. Multiply a salient weight channel by a factor greater than one before quantizing, so it occupies more of the grid and suffers proportionally less rounding error, and divide the corresponding activation channel by the same factor so the product is unchanged.

The scales are found by searching over a small parameter on calibration data. There is no backward pass, no reconstruction, and no weight updating, which makes AWQ fast to run.

Both algorithms consume the same two inputs, a trained weight matrix and a calibration set, and both aim at the same objective. They diverge in what they compute from the calibration data and what they change.

GPTQ computes second-order information, a full input correlation matrix per layer, and changes the weights themselves. Weights that have not yet been quantized get adjusted to absorb errors made earlier. The output is a checkpoint whose unquantized values would not match the original.

AWQ computes a first-order statistic, the average activation magnitude per channel, and changes only the scales. Weight values are rescaled by a per-channel factor that is exactly cancelled elsewhere, so the represented function is unchanged before rounding.

That difference explains their respective characters. GPTQ extracts more from the calibration set and depends on it more. AWQ extracts less and depends on it less, and runs faster because it never solves a linear system.

On reported 4-bit accuracy the two land close to each other.

Calibration sets are small. A typical run uses on the order of a hundred to a few hundred sequences of a couple of thousand tokens each, which is a few megabytes of text. The purpose is not to learn anything; it is to observe the distribution of activations the layer will encounter.

Because the set is small, its composition matters more than its size. Three failure modes recur.

Domain mismatch. Calibrating on general web text and deploying on code, legal documents or another language means the activation statistics you measured are not the ones you will see.

Length mismatch. Calibrating on short sequences and deploying with long contexts exercises different attention patterns and different activation ranges.

Format mismatch. Calibrating on raw text and deploying with a chat template, system prompts and tool-call syntax means the tokens driving your activations look nothing like the ones you measured.

The robust default is to sample calibration data from your own recorded traffic, in the exact format the model will receive it.

A consistent finding is that AWQ is less sensitive to the choice of calibration data than GPTQ. The mechanism explains why, and generalises beyond these two algorithms.

GPTQ fits a second-order reconstruction to the calibration activations. That is a rich model of the input distribution, and it extracts more signal when the calibration set is representative. It is also, for the same reason, capable of overfitting: the compensation it computes is tuned to correlations that may not hold on different traffic.

AWQ uses one number per channel, the average activation magnitude. That statistic is coarse, and coarse statistics transfer. Which channels carry large activations is a fairly stable property of a trained model, and it changes less across domains than the fine correlation structure does.

The general rule is the familiar bias-variance trade in a new setting. More expressive fitting on a small sample buys accuracy when the sample matches deployment and costs robustness when it does not.

If you cannot get representative calibration data, prefer the coarser method.

Everything so far is post-training quantization: the model is finished, and you are choosing how to represent it. At sufficiently low bit widths that stops being enough, and training has to know about the quantization.

Quantization-aware training inserts a simulated quantize-then-dequantize step into the forward pass, so the loss reflects the rounding the deployed model will do. The gradient problem is that rounding has zero derivative almost everywhere, which is solved with a straight-through estimator: use the rounded value in the forward pass, and pass the gradient through as if the rounding were the identity.

The model then learns weights that are robust to being rounded, rather than weights that happen to round well. This is the strongest option below 4 bits and the most expensive, because it requires a training run.

The practical ordering is unchanged by anything in this lesson: try round-to-nearest, then a calibration-based method, and reach for quantization-aware training only when measurement shows the post-training result is not good enough.

QLoRA is frequently filed alongside these methods and belongs in a different category, which is worth being precise about because the confusion leads to wrong expectations.

QLoRA, from Dettmers and colleagues, is a fine-tuning technique. The base model is frozen in 4-bit NF4. Small low-rank adapter matrices are trained in higher precision on top, and gradients flow through the frozen quantized base without updating it. Two supporting pieces make it fit: double quantization, which quantizes the quantization constants themselves to cut metadata overhead further, and paged optimizers to survive memory spikes.

What that buys is the ability to fine-tune a large model on a single accelerator. What it does not do is produce a better-quantized base model. The base is quantized with a straightforward NF4 scheme, and its errors remain.

So QLoRA and GPTQ or AWQ answer different questions. One is how to train cheaply on quantized weights. The other is how to make quantized weights faithful.

The choice is usually settled by deployment target and calibration data quality rather than by benchmark deltas, because at 4-bit weights the leading methods land close together.

Two habits matter more than the selection. Quantize with more than one method and compare on your own evaluation set, since the ranking is model-dependent and reverses between models. And record the calibration set alongside the checkpoint, because a quantized model is only reproducible if the data that produced it is.