Quantization & Optimization
Why lowering numerical precision barely hurts quality, the format landscape (bnb, GGUF, AWQ/GPTQ, FP8), and matching the format to the serving engine
Quantization & Optimization
TL;DR
Quantization stores model weights (and sometimes activations) at lower numerical precision to cut memory and increase speed, usually at a small accuracy cost. There is no single "correct" quantization format — bitsandbytes, GGUF, AWQ/GPTQ, and FP8 each trade off differently, and the right one depends less on the model than on what will actually serve it.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | Fine-Tuning with PEFT |
| You will understand | Why quantization works, the tradeoffs between formats, and how to pick one for the runtime you'll actually deploy on |
Why Lowering Precision Barely Hurts
A model's weights are floating-point numbers, and in practice they cluster in a fairly narrow range around zero rather than spreading evenly across the full range a 16- or 32-bit float can represent. Most of the precision a high bit-width buys you is spent distinguishing values that are already close together and contribute similarly to the output.
Quantization maps that narrow range onto a much smaller set of representable values — 256 levels for 8-bit, 16 for 4-bit — and stores an integer index plus a small per-block scale factor instead of the full float. The forward pass then dequantizes on the fly (or computes directly in low precision, depending on the format). The loss this introduces is real, but for most models and tasks it's small enough to be dominated by other sources of noise, right down to surprisingly aggressive 4-bit settings.
"Barely hurts" is a generalization, not a guarantee. Tasks that are unusually precision-sensitive — long chains of numerical reasoning, tasks near a decision boundary, some code generation — can show a real, measurable drop at low bit-widths. Always compare a quantized model against its full-precision baseline on your own evaluation set (see Evaluation) before assuming the loss is negligible for your task.
The Format Landscape
| Format | Precision | How it's produced | Typical use |
|---|---|---|---|
| bfloat16 / float16 | 16-bit | Native training/inference dtype, no quantization step | Default for training and most GPU inference |
| bitsandbytes 4-bit (NF4) | 4-bit | Quantized at load time, no calibration pass | QLoRA training, quick experimentation, inference inside transformers |
| GGUF | Mixed, typically 4–8 bit | Converted ahead of time for a specific target | CPU and edge inference via llama.cpp-family runtimes |
| AWQ / GPTQ | 3–4 bit | Calibrated post-training quantization — a short pass over sample data picks per-weight or per-group scales | Best accuracy retention at very low bit-widths, for GPU serving |
| FP8 | 8-bit float | Native support on recent GPU generations | Both training and inference, when hardware supports it — smaller accuracy loss than integer quantization at the same size |
Load-time vs calibrated quantization
Two different ways to get to a low bit-width
Load-time (bitsandbytes)
Quantizes the moment the model loads, with no separate preparation step — BitsAndBytesConfig(load_in_4bit=True, ...) and you're done. Fast to iterate with, which is exactly why it's the default for QLoRA. At very low bit-widths, quality can trail calibrated methods slightly, because there's no data-driven pass deciding which weights need more precision.
Calibrated (AWQ, GPTQ)
Runs a short calibration pass over representative sample data before fixing the quantization scheme, so weights or activations that matter more for real inputs can be preserved more carefully. Produces a separate, pre-quantized checkpoint you load directly — no load-time conversion cost, and generally the strongest quality at 3–4 bit. The cost is the calibration step itself and a less flexible workflow than "just load it quantized."
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="bfloat16",
)
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3.8-27B", quantization_config=bnb_config, device_map="auto"
)Match the Format to Where It Will Actually Run
This is the decision that matters more than which format is "best" in the abstract.
Pick the format from the serving target backward
| If you're serving with... | Reach for... |
|---|---|
transformers directly, in Python | bitsandbytes 4-bit — convenient, no separate conversion step |
llama.cpp or a CPU/edge runtime | GGUF — the format that ecosystem was built around |
| vLLM or TGI (high-throughput GPU serving) | AWQ, GPTQ, or FP8 — these engines have optimized kernels for them |
bitsandbytes inside transformers is a training and experimentation tool, not a production serving format. It's excellent for QLoRA and for quickly trying a model on limited hardware, but dedicated serving engines get meaningfully better throughput from formats built around their own kernels. Quantize once for the engine you'll actually run in production, rather than shipping the format that was easiest during development.
Beyond Weight Quantization: optimum and Hardware Export
optimum is HuggingFace's library for exporting models to hardware-specific runtimes — ONNX Runtime, OpenVINO, and others — for targets that aren't a standard GPU inference stack at all: CPU-only servers, specialized accelerators, or environments where a lean, dependency-light runtime matters more than raw throughput. It sits a layer below format choice: even a quantized model may need this kind of export to run efficiently outside a PyTorch/transformers environment.
diffusers pipelines support the same per-component quantization idea, letting you quantize just the heaviest component (usually the diffusion transformer) while leaving something like the text encoder at full precision — covered in Diffusers & Multimodal.
Concept Checks
Check yourself
Why does quantization lose so little accuracy for most models, given it's throwing away precision?
Because weights cluster in a narrow range, and most of a high bit-width's precision is spent distinguishing values that are already close together and contribute similarly to the output. Mapping that narrow range onto fewer representable levels, with a per-block scale factor, preserves most of the signal that actually matters for the forward pass.
A team quantizes a model to 4-bit with bitsandbytes for a code-generation task and quality drops noticeably. What should they check before switching format?
Whether they ever compared against the full-precision baseline on their own evaluation data — code generation is one of the more precision-sensitive tasks, so a real quality gap here isn't surprising. Before switching to a calibrated format like AWQ/GPTQ, it's worth confirming the gap is actually from quantization and measuring how much a calibrated method narrows it, rather than assuming any 4-bit format will fail equally.
A model is quantized with GGUF for local development, then the team moves to vLLM for production serving. What's the likely mistake here?
Carrying the development-time format into production without re-checking fit. GGUF is built for llama.cpp-family runtimes; vLLM's optimized kernels are built around formats like AWQ, GPTQ, or FP8, and won't get the same throughput benefit from a GGUF checkpoint. The format should be chosen for the serving engine, not carried over from whatever was convenient earlier in the pipeline.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Quantization | Storing weights/activations at lower precision to save memory and increase speed |
| NF4 | bitsandbytes' 4-bit format, load-time quantized, no calibration step |
| GGUF | The quantized format for llama.cpp-family CPU/edge inference |
| AWQ / GPTQ | Calibrated post-training quantization, strongest quality at 3–4 bit, built for GPU serving engines |
| FP8 | 8-bit floating point, increasingly used for both training and inference on recent GPUs |
| Load-time vs calibrated | Convenient and fast to iterate vs a data-driven pass for better low-bit quality |
optimum | Hardware-specific export (ONNX, OpenVINO) for non-standard serving targets |
| Format-matches-engine rule | Quantize once for the serving engine you'll actually run in production |
Next
A quantized model is only as trustworthy as the numbers you use to compare it against the alternative: Evaluation.
Fine-Tuning with PEFT
Full fine-tuning vs LoRA vs QLoRA — the mechanics, the memory math, and how to train an adapter without wrecking the base model
Evaluation
Metrics with the evaluate library, why a leaderboard score isn't your score, building a held-out test set, and LLM-as-judge for generative tasks