Quantization for Edge
GGUF quant types, AWQ/GPTQ, bitsandbytes, and BitNet — how to shrink a small model further, and which format fits which device
Quantization for Edge
TL;DR
A trained model ships in 16-bit precision. Almost nothing that runs on a phone, laptop, or browser tab wants to carry that weight. GGUF quantization (used by llama.cpp) is the default for CPU and edge; AWQ/GPTQ are the calibrated alternative for GPU serving; BitNet is a training-time decision, not something you can bolt on afterward. Picking the wrong one for your target device is the single most common edge-deployment mistake.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | Fine-Tuning Small Models |
| You will understand | Every mainstream quantization format for small models, and how to pick one by target device |
Why Quantize at All
A 1.2B-parameter model in fp16 is roughly 2.4 GB — two bytes per parameter. That's already borderline for a phone that also has to run everything else on the device, and it's real bandwidth to download over a mobile connection. Quantization stores each weight in fewer bits, trading a small, usually-recoverable amount of accuracy for a much smaller file and — because decoding is memory-bandwidth-bound, not compute-bound — genuinely faster inference too.
model.safetensors (fp16, ~2.4 GB for a 1.2B model)
│ quantize
▼
model-Q4_K_M.gguf (~700 MB) ← same model, comfortable on a phoneQuantization usually speeds up inference, not just shrinks the file. Since generating each token means reading the whole weight matrix from memory, a smaller weight matrix means less data to move per token — this is why a 4-bit model is often noticeably faster to generate from than the same model in fp16, on the same hardware.
GGUF: The Edge/CPU Default
GGUF is the file format used by llama.cpp and its compatible runtimes (Ollama, LM Studio, and others build on top of it). It bundles the quantized weights, tokenizer, and metadata into one file that a C/C++ runtime can load with no Python, no CUDA, and no framework dependency at all.
The quant types that matter
| Type | Bits | What it trades | Use when |
|---|---|---|---|
| Q8_0 | 8-bit | Near-lossless vs fp16, still ~2x smaller | You have the memory and want minimal quality risk |
| Q5_K_M | ~5-bit (mixed) | A step above Q4 in quality, larger file | Quality matters more than squeezing every megabyte |
| Q4_K_M | ~4-bit (mixed) | The default sweet spot — small, fast, minor quality loss | The right starting point for almost every edge deployment |
| Q3 / Q2 | ~3-bit / ~2-bit | Meaningful quality loss, smallest files | Only when the device genuinely cannot fit anything bigger |
The _K_M suffix means the quantizer doesn't apply one bit-width uniformly across the whole model — it's a K-quant: different tensor types get different precision based on how sensitive they are to quantization error (attention weights are typically more sensitive than some feed-forward weights), and M (medium) picks a specific mix of those precisions. _S (small) and _L (large) variants shift that mix toward smaller or higher-quality.
Start at Q4_K_M, not the smallest option that fits. It's the level the ecosystem has converged on as "small enough to matter, rarely small enough to notice the loss." Drop lower only after you've actually measured a quality regression you can tolerate — see Evaluation for SLMs for how to check.
Why memory-mapping matters
GGUF is designed to be memory-mapped rather than fully loaded into RAM before use. The operating system pages the file in from disk on demand as the model runs, instead of llama.cpp reading the whole multi-hundred-megabyte file up front. On a constrained device this is the difference between a near-instant cold start and a multi-second stall before the first token — and it's also why GGUF models tolerate memory pressure more gracefully than a runtime that insists on the whole model resident in RAM at once.
llama.cpp runs on CPU (with AVX/AVX2/AVX512 acceleration where available) and GPU (CUDA and others), and exposes an OpenAI-compatible API server — so the same GGUF file can serve a phone app, a desktop tool, or a small self-hosted API without a rewrite. Hugging Face Inference Endpoints can also deploy a llama.cpp container directly from a GGUF-format model repo, if you want the same artifact behind a hosted endpoint rather than fully on-device.
AWQ and GPTQ: The GPU-Serving Alternative
GGUF targets CPU/edge. AWQ and GPTQ target the opposite end: GPU serving engines like vLLM and TGI, where the goal is maximum throughput across many concurrent requests rather than running on a single constrained device.
Both are calibrated — unlike GGUF's block-based quantization, AWQ and GPTQ run a calibration pass over a small sample of representative data first, using that to decide which weights are most sensitive and protect them more carefully during quantization. This generally gets better quality at very low bit-widths (3-4 bit) than an uncalibrated approach, at the cost of needing that calibration step at all.
GGUF vs AWQ/GPTQ — pick by where the model runs, not by preference
GGUF
RecommendedCPU and edge devices, llama.cpp-family runtimes, no calibration step needed, memory-mapped for fast cold starts. The right default for on-device deployment.
AWQ / GPTQ
GPU serving engines (vLLM, TGI), calibrated for strong quality at 3-4 bit, built for throughput across many requests rather than single-device deployment.
bitsandbytes: The Development-Time Tool
bitsandbytes quantizes at load time — 8-bit or 4-bit, no calibration pass, just pass a config and it converts weights as they load into transformers. That convenience is exactly why it's the right tool during training and experimentation (it's what makes QLoRA practical — see Fine-Tuning Small Models) and usually the wrong tool for the final shipped edge artifact: it's slower at inference than GPTQ or plain fp16 serving, according to the serving engines that support it, because it quantizes on the fly rather than shipping pre-quantized weights.
bitsandbytes is for the lab, GGUF/AWQ/GPTQ are for shipping. A model you quantized with bitsandbytes to fit it on your training GPU is not the artifact you want to hand to a phone app — convert to GGUF (or AWQ/GPTQ for GPU serving) as a distinct step before deployment.
BitNet: Quantization You Can't Add Later
Every format above quantizes an already-trained fp16 checkpoint. BitNet is different in a way that trips people up constantly: it replaces the model's linear layers with BitLinear layers that constrain weights to just three values — -1, 0, and 1 (ternary precision) — and quantizes activations to 8-bit.
The mechanism, briefly:
- Compute the average absolute value of the weight matrix — that's the scale.
- Divide weights by that scale, round, and clamp to [-1, 1].
- Rescale to continue the forward pass in full precision internally, while the stored weight is one of 1.
The gotcha that matters most on this page
BitNet models cannot be quantized on the fly. This is a Quantization-Aware Training (QAT) technique — the ternary weights are what the model is trained (or fine-tuned) with from the very start, not something you apply to a finished fp16 checkpoint the way you'd convert to GGUF. If you didn't decide on BitNet before or during training, you cannot retrofit it afterward. Plan for it at the pretraining/fine-tuning stage, not at the deployment stage.
transformers has native BitNet support for both training and inference. The payoff, when it fits your situation, is real: ternary weights are dramatically smaller and cheaper to compute with than even 4-bit quantized weights — but only for models actually built this way from the start.
Choosing Bit-Width by Target Device
A practical starting point
Phone / embedded
RecommendedSmallest model that clears your accuracy bar, GGUF Q4_K_M (or lower only if RAM genuinely forces it). Prioritize file size and cold-start time as hard as accuracy.
Laptop / desktop app
GGUF Q5_K_M or Q8_0 is usually comfortable — more RAM and storage headroom than mobile, so there's less reason to push to the most aggressive quant available.
Browser (WebGPU)
Whatever quantized ONNX/GGUF export your runtime (e.g. transformers.js) supports well — covered in Inference Engines & Runtimes.
Server GPU, high throughput
AWQ, GPTQ, or fp8 through vLLM/TGI — optimized for concurrent request throughput, not single-device footprint.
Concept Checks
Check yourself
You quantized a model with bitsandbytes to fit training on your GPU. Can you ship that exact artifact to a phone app?
Not as the final artifact — bitsandbytes quantizes at load time and is optimized for training/experimentation convenience, not edge inference speed. Convert to GGUF (or another edge-appropriate format) as a separate step before shipping to a device.
A colleague wants to take your team's existing fine-tuned fp16 checkpoint and 'just BitNet-quantize it' before deployment next week. What's wrong with that plan?
BitNet is a Quantization-Aware Training technique, not a post-hoc quantization format like GGUF or AWQ. The ternary weights have to be what the model was trained with — you cannot take a finished fp16 checkpoint and convert it to BitNet the way you'd convert it to Q4_K_M. That decision needed to be made before or during training/fine-tuning, not the week before deployment.
Why does GGUF's memory-mapping matter more on a phone than on a beefy server?
A constrained device benefits from the OS paging the model in from disk on demand rather than loading the whole file into RAM before the first token — this gets cold-start time down and lets the model tolerate memory pressure from other apps. A server with abundant RAM has less to gain from this property, since it can usually just load the whole model into memory up front anyway.
Why might AWQ/GPTQ produce better quality than GGUF at the same bit-width?
AWQ and GPTQ run a calibration pass over representative sample data before quantizing, which lets them identify and protect the weights most sensitive to quantization error. GGUF's K-quants use a fixed heuristic mix of precisions by tensor type rather than a calibration pass, which is faster and simpler but can be less precisely tuned to a specific model's actual sensitivity pattern.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| GGUF | The llama.cpp-family quantization/file format, edge/CPU-oriented, memory-mapped |
| Q4_K_M | The default sweet spot GGUF quant level — small, fast, minor quality loss |
| K-quant | Mixed per-tensor-type precision rather than one uniform bit-width |
| AWQ / GPTQ | Calibrated post-training quantization, GPU-serving-oriented |
| bitsandbytes | Load-time quantization, best for training/experimentation, not final edge shipping |
| BitNet | Ternary (1) weights via Quantization-Aware Training — a training-time, not deployment-time, decision |
| Memory-mapping | Paging the model in from disk on demand, key to fast cold starts on constrained devices |
Next
Choosing the right quantized artifact only matters once something can actually run it: Inference Engines & Runtimes.
Fine-Tuning Small Models
Why full fine-tuning is back on the table at small scale, why catastrophic forgetting hits harder, and the extreme-efficiency PEFT methods worth knowing
Inference Engines & Runtimes
llama.cpp, transformers.js with WebGPU, mobile-native runtimes, and vLLM/TGI — matching the runtime to where the model actually has to run