Quantization From First Principles
The exact math behind mapping float weights to integers, worked numerically, plus symmetric vs asymmetric, per-tensor vs per-channel, and the straight-through estimator behind QAT
Quantization From First Principles
TL;DR
Quantization maps a continuous range of float weights onto a small set of integers via a scale and zero-point, and the reconstruction error is usually small because trained weights cluster near zero rather than spreading uniformly across their range. Quantization-aware training simulates this rounding during training itself, using a trick called the straight-through estimator to let gradients flow through an operation that would otherwise have none.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~19 minutes |
| Prerequisites | Fine-Tuning & Transfer Learning |
| You will understand | The exact math of linear quantization, why it works well in practice, and how QAT trains through a non-differentiable rounding step |
The Core Mechanism: Linear (Affine) Quantization
The idea: pick a scale and a zero-point that map a float range onto an integer range, then round.
scale = (float_max - float_min) / (quant_max - quant_min)
zero_point = round(quant_min - float_min / scale)
quantize: q = round(x / scale) + zero_point
dequantize: x̂ = (q - zero_point) * scaleA fully worked numeric example
Take five real-looking trained weights: [-0.82, -0.11, 0.03, 0.40, 0.95], quantizing to signed 8-bit integers (range -128 to 127).
float_min = -0.82, float_max = 0.95
scale = (0.95 - (-0.82)) / (127 - (-128)) = 1.77 / 255 ≈ 0.006941
zero_point = round(-128 - (-0.82 / 0.006941)) = round(-128 + 118.14) ≈ -10
Quantizing each weight:
-0.82 → round(-0.82/0.006941) + (-10) = round(-118.14) - 10 = -118 - 10 = -128
-0.11 → round(-15.85) - 10 = -16 - 10 = -26
0.03 → round(4.32) - 10 = 4 - 10 = -6
0.40 → round(57.64) - 10 = 58 - 10 = 48
0.95 → round(136.9) - 10 = 137 - 10 = 127
Dequantizing back:
-128 → (-128 - (-10)) * 0.006941 = -0.8189 (original: -0.82, error: 0.0011)
-26 → (-26 - (-10)) * 0.006941 = -0.1110 (original: -0.11, error: 0.0010)
-6 → ( -6 - (-10)) * 0.006941 = 0.0278 (original: 0.03, error: 0.0022)
48 → ( 48 - (-10)) * 0.006941 = 0.4025 (original: 0.40, error: 0.0025)
127 → (127 - (-10)) * 0.006941 = 0.9508 (original: 0.95, error: 0.0008)Every reconstructed value is within about 0.002 of the original — a small error relative to the values themselves, and the reason quantized models usually lose only a little accuracy rather than breaking outright.
Symmetric vs Asymmetric Quantization
Two ways to place the range
Symmetric
RecommendedAssumes the range is centered at zero: scale = max(|x|) / quant_max, and zero_point = 0 always. Simpler math, no zero-point term to track, and works well for weight distributions — which are typically naturally roughly zero-centered after training with weight decay.
Asymmetric
Uses the full [min, max] range with a nonzero zero-point, as in the worked example above. Needed for distributions that aren't centered at zero — the classic case is post-ReLU activations, which are entirely non-negative, so a symmetric scheme centered at zero would waste half its representable range on negative values that never occur.
def symmetric_quantize(x, bits=8):
qmax = 2**(bits - 1) - 1
scale = x.abs().max() / qmax
q = torch.round(x / scale).clamp(-qmax - 1, qmax)
return q, scale # no zero_point needed — it's always 0Per-Tensor vs Per-Channel Quantization
A single scale for an entire weight matrix is simple, but weight matrices don't have uniform magnitude across all their rows.
Per-tensor: one scale for the whole matrix W (shape d_out × d_in)
Per-channel: one scale PER ROW of W — d_out separate scalesDifferent output channels (rows) of a weight matrix often have meaningfully different natural magnitude ranges — one row might have values spanning ±0.1, another ±2.0. A single per-tensor scale has to accommodate the widest row, which wastes precision on every narrower row: values that only span ±0.1 get quantized using a scale sized for ±2.0, throwing away most of their available resolution. Per-channel quantization gives each row its own scale, so each one uses its full available precision regardless of what the others look like.
This is exactly why practical quantization formats — the K-quants used in GGUF, covered in the Small Language Models track — go even further than per-channel, using different bit-widths and block-level scales tuned to how sensitive different parts of a tensor actually are to precision loss.
Why the Error Is Usually Small in Practice
Trained neural network weights aren't spread uniformly across their range — after training with weight decay (see Regularization & Generalization), most weights cluster fairly densely near zero, with progressively fewer values as magnitude increases, roughly Gaussian-shaped.
A quantization scheme that allocates its limited discrete levels uniformly across the observed range still ends up representing the dense, near-zero region reasonably well, simply because that's where most of the actual values are — the coarser representation falls mostly on the sparse tails, where a few extra millivolts of error matters less to the overall computation. This is the deeper reason simple uniform quantization works as well as it does, and it's also exactly what motivates non-uniform schemes: if you know where the density actually concentrates, you can allocate precision there even more deliberately than uniform quantization does automatically.
Quantization-Aware Training and the Straight-Through Estimator
Post-training quantization (everything above) quantizes an already-trained model. QAT instead simulates the rounding during training, so the model learns weights that are robust to the precision loss it will actually experience at deployment.
The obstacle: round() has a derivative of zero almost everywhere (it's a staircase function), and is undefined at the jumps. Plugged directly into backpropagation, this would kill the gradient entirely — no learning signal could pass through a rounding operation.
class FakeQuantize(torch.autograd.Function):
@staticmethod
def forward(ctx, x, scale):
return torch.round(x / scale) * scale # the actual quantization effect
@staticmethod
def backward(ctx, grad_output):
return grad_output, None # pretend rounding was the identity functionThe straight-through estimator (STE) is exactly this trick: use the real, rounded value on the forward pass, but pretend the rounding operation was the identity function on the backward pass, letting gradients flow through unchanged as if no rounding had happened at all. It's mathematically "wrong" — the true gradient of a step function is zero or undefined — but it works remarkably well in practice, because it still points training in roughly the right direction even though the exact gradient it's using isn't technically correct.
This is precisely the mechanism BitNet depends on to train with ternary (1) weights from the start — see the practical coverage in Quantization for Edge — since ternary rounding is an even more extreme non-differentiable operation than ordinary int8 rounding, and STE is what makes gradient-based training possible through it at all.
Concept Checks
Check yourself
In the worked numeric example, why isn't the dequantized value exactly equal to the original float weight?
Because quantization maps a continuous range of float values onto a much smaller discrete set of integers, and rounding to the nearest representable integer during the quantize step necessarily loses some information — the dequantized value can only reconstruct what that discrete integer represents, not the exact original float. The size of this loss is bounded by roughly half the scale value, which is why a well-chosen scale keeps the error small.
Why would per-tensor quantization waste precision on a weight matrix where different rows have very different natural magnitude ranges?
Because a single scale has to be sized to accommodate the widest-range row, and that same scale then gets applied to every other row too — a row with a much narrower range ends up using only a small fraction of the available integer levels, throwing away resolution it could otherwise have used. Per-channel quantization fixes this by giving each row its own scale sized to its own actual range.
Why can't you just backpropagate through round() directly, and what does the straight-through estimator do instead?
Because round() is a staircase function whose true derivative is zero almost everywhere and undefined at the jumps — plugged directly into backpropagation, it would block virtually all gradient signal from passing through. The straight-through estimator sidesteps this by using the real rounded value on the forward pass but treating the rounding operation as if it were the identity function on the backward pass, letting an approximate-but-useful gradient flow through anyway.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Scale / zero-point | The two numbers that map a float range onto an integer range |
| Quantize / dequantize | Round-and-shift to integers, then reverse the mapping to approximate the original |
| Symmetric quantization | Zero-centered, no zero-point needed — fits typical weight distributions |
| Asymmetric quantization | Full range with a zero-point — needed for skewed distributions like post-ReLU activations |
| Per-channel quantization | A separate scale per row, avoiding one row's wide range degrading another's precision |
| Why error is usually small | Trained weights cluster near zero, so most values are well-represented even by uniform quantization |
| QAT | Training with quantization's rounding simulated, so the model learns weights robust to it |
| Straight-through estimator | Uses the real rounded value forward, but treats rounding as identity on the backward pass |
Next
With the model shrunk, the next page covers a different compression strategy — training a smaller model to imitate a larger one directly: Knowledge Distillation Mechanics.
Fine-Tuning & Transfer Learning
Why transfer learning works, how layer freezing works mechanically, and the full derivation of LoRA's low-rank update
Knowledge Distillation Mechanics
The full distillation loss — temperature-scaled soft targets, KL divergence, and why the loss needs a T² correction term